Jul 30 20:57:08.104882 ip-10-0-143-139 systemd[1]: Starting Kubernetes Kubelet...
Jul 30 20:57:08.553577 ip-10-0-143-139 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 20:57:08.553577 ip-10-0-143-139 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 30 20:57:08.553577 ip-10-0-143-139 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 20:57:08.553577 ip-10-0-143-139 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 30 20:57:08.553577 ip-10-0-143-139 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 30 20:57:08.554666 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.554581    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 30 20:57:08.558946 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558931    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.558946 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558946    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558951    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558954    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558957    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558961    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558963    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558966    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558969    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558972    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558975    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558978    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558980    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558984    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558988    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558991    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558994    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.558998    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559002    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559013    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.559007 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559016    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559019    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559023    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559026    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559028    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559031    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559033    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559036    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559038    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559041    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559043    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559046    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559049    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559051    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559054    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559056    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559059    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559061    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559064    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559066    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.559551 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559068    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559071    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559073    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559075    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559078    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559080    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559083    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559085    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559088    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559090    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559093    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559096    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559098    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559100    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559104    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559106    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559109    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559111    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559114    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559116    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559119    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.560088 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559121    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559124    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559126    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559129    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559131    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559134    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559136    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559139    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559141    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559144    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559146    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559149    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559153    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559156    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559159    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559162    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559164    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559167    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559170    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.560599 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559172    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559175    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559177    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559180    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559182    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559185    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559189    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559191    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559194    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559197    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559199    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559202    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559204    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559207    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559209    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559211    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559214    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559216    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559219    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.561105 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559914    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559920    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559923    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559926    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559928    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559932    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559934    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559937    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559940    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559942    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559945    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559949    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559951    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559955    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559959    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559962    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559965    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559968    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559971    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.561583 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559974    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559976    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559979    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559981    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559984    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559987    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559989    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559992    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559995    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.559998    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560000    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560003    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560007    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560010    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560013    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560016    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560019    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560023    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560026    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.562067 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560030    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560033    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560035    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560038    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560041    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560043    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560046    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560049    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560052    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560054    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560056    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560059    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560061    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560064    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560067    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560069    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560071    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560074    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560076    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560079    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.562527 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560082    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560085    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560087    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560089    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560092    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560095    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560097    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560100    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560103    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560105    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560108    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560110    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560112    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560115    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560118    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560120    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560123    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560126    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560128    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560131    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.563035 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560133    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560136    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560139    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560141    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560143    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560148    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560151    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560153    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560156    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560158    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560160    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560163    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560165    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560167    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560170    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560172    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560174    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560177    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560179    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560182    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.563525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560184    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560541    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560546    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560549    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560552    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560555    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560558    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560560    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560579    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560583    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560586    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560588    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560591    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560593    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560596    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560598    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560601    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560606    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560609    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.564055 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560613    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560616    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560619    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560622    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560624    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560627    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560629    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560632    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560634    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560637    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560639    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560642    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560645    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560647    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560650    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560653    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560656    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560658    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560660    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.564511 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560664    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560666    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560669    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560672    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560674    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560677    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560680    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560682    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560685    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560687    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560690    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560695    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560698    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560700    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560703    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560705    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560708    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560710    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560713    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560718    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.564979 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560721    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560724    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560726    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560729    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560732    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560734    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560737    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560739    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560741    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560744    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560746    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560749    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560752    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560755    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560758    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560760    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560763    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560765    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560768    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560770    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.565525 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560773    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560775    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560777    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560781    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560783    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560786    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560788    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560791    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560793    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560796    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560799    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560801    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560803    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560806    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560809    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560811    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560814    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560816    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560818    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560822    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.566042 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560825    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.560827    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560898    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560905    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560912    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560916    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560920    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560923    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560927    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560931    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560934    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560937    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560941    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560944    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560947    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560951    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560954    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560957    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560959    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560962    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560967    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560970    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560973    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560975    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 30 20:57:08.566528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560979    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560982    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560985    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560988    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560991    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560994    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.560997    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561003    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561006    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561009    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561013    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561016    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561019    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561022    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561024    2583 flags.go:64] FLAG: --enable-server="true"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561028    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561032    2583 flags.go:64] FLAG: --event-burst="100"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561035    2583 flags.go:64] FLAG: --event-qps="50"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561038    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561041    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561044    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561047    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561050    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561054    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561057    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 30 20:57:08.567138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561060    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561063    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561066    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561069    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561072    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561074    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561077    2583 flags.go:64] FLAG: --feature-gates=""
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561081    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561084    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561087    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561090    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561093    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561095    2583 flags.go:64] FLAG: --help="false"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561098    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561103    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561106    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561109    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561113    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561116    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561119    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561122    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561125    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561128    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 30 20:57:08.567773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561131    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561134    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561137    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561140    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561143    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561145    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561148    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561152    2583 flags.go:64] FLAG: --lock-file=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561155    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561157    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561160    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561165    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561168    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561171    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561173    2583 flags.go:64] FLAG: --logging-format="text"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561176    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561179    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561182    2583 flags.go:64] FLAG: --manifest-url=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561184    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561188    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561191    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561195    2583 flags.go:64] FLAG: --max-pods="110"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561198    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561201    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561204    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 30 20:57:08.568351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561207    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561210    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561213    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561215    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561222    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561225    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561228    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561231    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561234    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561239    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561242    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561245    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561247    2583 flags.go:64] FLAG: --port="10250"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561250    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561254    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af1f05275f42606a"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561257    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561260    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561263    2583 flags.go:64] FLAG: --register-node="true"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561266    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561269    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561272    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561274    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561277    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561281    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561284    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 30 20:57:08.568982 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561287    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561289    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561292    2583 flags.go:64] FLAG: --runonce="false"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561295    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561298    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561301    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561306    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561308    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561311    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561314    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561317    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561320    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561323    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561326    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561328    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561331    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561334    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561337    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561342    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561345    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561348    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561352    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561355    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561357    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561360    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 30 20:57:08.569594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561363    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561366    2583 flags.go:64] FLAG: --v="2"
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561369    2583 flags.go:64] FLAG: --version="false"
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561374    2583 flags.go:64] FLAG: --vmodule=""
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561382    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.561385    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561497    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561500    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561503    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561506    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561510    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561512    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561515    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561520    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561523    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561525    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561528    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561530    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561532    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561535    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561538    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.570206 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561540    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561543    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561545    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561548    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561551    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561553    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561556    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561558    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561575    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561578    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561582    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561586    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561590    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561593    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561597    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561600    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561603    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561606    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561609    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.570765 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561612    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561614    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561617    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561619    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561622    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561626    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561629    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561631    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561634    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561637    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561639    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561642    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561644    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561646    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561649    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561652    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561654    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561657    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561660    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561662    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.571226 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561667    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561669    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561671    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561674    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561676    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561679    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561681    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561684    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561686    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561689    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561691    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561693    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561696    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561698    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561701    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561703    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561706    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561709    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561712    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561714    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.571722 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561717    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561721    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561723    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561726    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561728    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561731    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561733    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561736    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561738    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561741    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561747    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561750    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561754    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561756    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561758    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561761    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561764    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561766    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561769    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561771    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.572216 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561773    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561776    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561778    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561781    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.561783    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.562324    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.569260    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.569275    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.570114    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.570124    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570184    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570189    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570192    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570195    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570199    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.572753 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570202    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570205    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570208    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570211    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570214    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570219    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570221    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570225    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570228    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570231    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570234    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570237    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570240    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570243    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570245    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570248    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570250    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570253    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570255    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570258    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.573131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570260    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570263    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570265    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570268    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570271    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570274    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570277    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570279    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570283    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570286    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570289    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570295    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570297    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570300    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570302    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570305    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570307    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570309    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570312    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.573630 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570315    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570318    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570321    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570324    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570326    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570329    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570331    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570333    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570336    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570338    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570341    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570343    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570345    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570348    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570350    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570353    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570356    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570359    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570361    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570364    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.574163 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570366    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570369    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570371    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570374    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570376    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570379    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570381    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570383    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570386    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570388    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570391    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570393    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570397    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570404    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570407    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570409    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570413    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570416    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570419    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.574679 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570422    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570424    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570427    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570430    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570432    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570434    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570437    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570440    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570443    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570445    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570448    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570450    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570453    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570455    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.575130 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570458    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.570463    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570589    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570594    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570597    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570600    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570603    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570606    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570609    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570613    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570616    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570619    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570621    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570624    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570627    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 30 20:57:08.575466 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570629    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570631    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570634    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570636    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570639    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570641    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570643    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570646    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570648    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570650    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570653    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570655    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570658    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570660    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570663    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570666    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570668    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570670    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570673    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570675    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 30 20:57:08.575845 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570678    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570680    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570682    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570685    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570687    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570690    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570692    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570695    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570699    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570702    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570704    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570707    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570710    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570713    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570716    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570718    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570721    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570723    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570726    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570728    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 30 20:57:08.576347 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570731    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570733    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570736    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570739    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570741    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570744    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570746    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570749    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570751    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570754    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570756    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570760    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570763    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570766    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570769    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570772    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570775    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570778    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570780    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570783    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 30 20:57:08.576912 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570785    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570788    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570790    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570792    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570795    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570798    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570800    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570803    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570805    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570808    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570810    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570812    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570815    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570817    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570820    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570822    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570825    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570827    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570829    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570832    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 30 20:57:08.577393 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570834    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570837    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570839    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570842    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570844    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:08.570846    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.570851    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.570950    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.575262    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.576766    2583 server.go:1019] "Starting client certificate rotation"
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.576864    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 20:57:08.577898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.576897    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 30 20:57:08.580148 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.580133    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 20:57:08.586205 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.586179    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 30 20:57:08.600127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.600106    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 30 20:57:08.602197 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.602181    2583 log.go:25] "Validated CRI v1 image API"
Jul 30 20:57:08.603014 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.602994    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 30 20:57:08.607239 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.607213    2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 beada6ba-66de-42d5-bb8d-1c1c74bbe301:/dev/nvme0n1p4 ed9d642f-64ca-4613-a50d-165f61b97b0e:/dev/nvme0n1p3]
Jul 30 20:57:08.607311 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.607237    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 30 20:57:08.610858 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.610838    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 20:57:08.613436 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.613329    2583 manager.go:216] Machine: {Timestamp:2026-07-30 20:57:08.611784684 +0000 UTC m=+0.377746554 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097775 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec218ea9d8bd51757c37899199c764ef SystemUUID:ec218ea9-d8bd-5175-7c37-899199c764ef BootID:fd6a86f0-df58-4dcc-b0c7-b09857d912c6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:b3:00:2b:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:b3:00:2b:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:f7:94:a7:bd:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 30 20:57:08.613436 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.613430    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 30 20:57:08.613559 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.613508    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 30 20:57:08.617114 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.617089    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 30 20:57:08.617242 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.617117    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 30 20:57:08.617994 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.617983    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 30 20:57:08.618026 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.617997    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 30 20:57:08.618026 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.618004    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 20:57:08.618026 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.618017    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 30 20:57:08.619406 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.619395    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 30 20:57:08.623004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.622992    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 20:57:08.623124 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623113    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 30 20:57:08.623155 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623147    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 30 20:57:08.623182 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623157    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 30 20:57:08.623182 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623171    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 30 20:57:08.623182 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623181    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 30 20:57:08.623262 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623189    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 30 20:57:08.623621 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623607    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 20:57:08.623677 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.623633    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 30 20:57:08.627095 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.627079    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 30 20:57:08.628736 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.628720    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 30 20:57:08.628812 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.628758    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 30 20:57:08.630032 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630011    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630039    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630050    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630060    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630073    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630088    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 30 20:57:08.630100 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630100    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 30 20:57:08.630264 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630110    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 30 20:57:08.630264 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630121    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 30 20:57:08.630264 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630130    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 30 20:57:08.630264 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630143    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 30 20:57:08.630264 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.630153    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 30 20:57:08.631153 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.631132    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 30 20:57:08.631153 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.631151    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 30 20:57:08.634542 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.634527    2583 server.go:1268] "Started kubelet"
Jul 30 20:57:08.634656 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.634609    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 30 20:57:08.635023 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.634963    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 30 20:57:08.635138 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.635052    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 30 20:57:08.635198 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.635136    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 30 20:57:08.635521 ip-10-0-143-139 systemd[1]: Started Kubernetes Kubelet.
Jul 30 20:57:08.636060 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.635973    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 30 20:57:08.636060 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.636012    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 30 20:57:08.636427 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.636387    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 30 20:57:08.636842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.636830    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 30 20:57:08.641193 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.641161    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 30 20:57:08.642799 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.641632    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18c72ccb39bf16a9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-07-30 20:57:08.634498729 +0000 UTC m=+0.400460659,LastTimestamp:2026-07-30 20:57:08.634498729 +0000 UTC m=+0.400460659,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Jul 30 20:57:08.643160 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.643138    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 30 20:57:08.643160 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.643160    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 30 20:57:08.643497 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.643305    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:08.643992 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.643976    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 30 20:57:08.643992 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.643992    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.643990    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644022    2583 factory.go:55] Registering systemd factory
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644032    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644084    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644095    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 30 20:57:08.644127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644103    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 30 20:57:08.644377 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644266    2583 factory.go:153] Registering CRI-O factory
Jul 30 20:57:08.644377 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644283    2583 factory.go:223] Registration of the crio container factory successfully
Jul 30 20:57:08.644377 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644307    2583 factory.go:103] Registering Raw factory
Jul 30 20:57:08.644377 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644320    2583 manager.go:1196] Started watching for new ooms in manager
Jul 30 20:57:08.644913 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.644899    2583 manager.go:349] Starting recovery of all containers
Jul 30 20:57:08.649373 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.649345    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 30 20:57:08.649554 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.649529    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 30 20:57:08.655040 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.655018    2583 manager.go:354] Recovery completed
Jul 30 20:57:08.660753 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.660740    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.662755 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.662731    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.662828 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.662768    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.662828 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.662780    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.663327 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.663313    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.663399 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.663312    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 30 20:57:08.663399 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.663347    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 30 20:57:08.663399 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.663366    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 30 20:57:08.664118 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664106    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.664168 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664129    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.664168 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664138    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.664384 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664369    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kncgv"
Jul 30 20:57:08.664905 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664886    2583 policy_none.go:49] "None policy: Start"
Jul 30 20:57:08.665001 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664908    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 30 20:57:08.665001 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.664924    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 30 20:57:08.665001 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.664931    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18c72ccb3b6e3cbd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-139.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-07-30 20:57:08.662754493 +0000 UTC m=+0.428716363,LastTimestamp:2026-07-30 20:57:08.662754493 +0000 UTC m=+0.428716363,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Jul 30 20:57:08.666810 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.666798    2583 policy_none.go:47] "Start"
Jul 30 20:57:08.671813 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.671792    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kncgv"
Jul 30 20:57:08.673090 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.673003    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18c72ccb3b6e8b1b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-139.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-07-30 20:57:08.662774555 +0000 UTC m=+0.428736425,LastTimestamp:2026-07-30 20:57:08.662774555 +0000 UTC m=+0.428736425,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Jul 30 20:57:08.674945 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.674887    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-139.ec2.internal.18c72ccb3b6eaf5a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-139.ec2.internal,UID:ip-10-0-143-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-139.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-139.ec2.internal,},FirstTimestamp:2026-07-30 20:57:08.662783834 +0000 UTC m=+0.428745704,LastTimestamp:2026-07-30 20:57:08.662783834 +0000 UTC m=+0.428745704,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-139.ec2.internal,}"
Jul 30 20:57:08.698117 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698099    2583 manager.go:337] "Starting Device Plugin manager"
Jul 30 20:57:08.698205 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.698129    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 30 20:57:08.698205 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698138    2583 server.go:87] "Starting device plugin registration server"
Jul 30 20:57:08.698365 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698354    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 30 20:57:08.698418 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698370    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 30 20:57:08.698509 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698494    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 30 20:57:08.698648 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698631    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 30 20:57:08.698648 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.698646    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 30 20:57:08.699070 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.699049    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 30 20:57:08.699132 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.699104    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:08.769165 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.769127    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 30 20:57:08.770515 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.770484    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 30 20:57:08.770634 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.770535    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 30 20:57:08.770634 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.770560    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 30 20:57:08.770634 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.770621    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 30 20:57:08.773801 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.773781    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 20:57:08.799182 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.799161    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.800124 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.800109    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.800188 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.800138    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.800188 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.800150    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.800188 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.800177    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.810191 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.810144    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.810191 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.810164    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-139.ec2.internal\": node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:08.845715 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.845693    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:08.870734 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.870708    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"]
Jul 30 20:57:08.870813 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.870770    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.872183 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.872159    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.872291 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.872194    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.872291 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.872210    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.873677 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.873665    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.873771 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.873758    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.873818 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.873784    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.874502 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874484    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.874598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874517    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.874598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874520    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.874598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874532    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.874598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874540    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.874598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.874550    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.875943 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.875927    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.875987 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.875963    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:08.876699 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.876684    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:08.876802 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.876713    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:08.876802 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.876729    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:08.893126 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.893107    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.896868 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.896852    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-139.ec2.internal\" not found" node="ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.946348 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:08.946320    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:08.946475 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.946368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.946475 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.946398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:08.946475 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:08.946418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d952314a66eed41f4a061771aac559f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"2d952314a66eed41f4a061771aac559f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046464 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.046407    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.046642 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046642 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046642 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046547    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d952314a66eed41f4a061771aac559f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"2d952314a66eed41f4a061771aac559f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046642 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d952314a66eed41f4a061771aac559f-config\") pod \"kube-apiserver-proxy-ip-10-0-143-139.ec2.internal\" (UID: \"2d952314a66eed41f4a061771aac559f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046642 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.046816 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.046616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba74c42e79d4ab3ec2af27545ae5c8d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal\" (UID: \"ba74c42e79d4ab3ec2af27545ae5c8d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.147271 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.147201    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.195428 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.195398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.199883 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.199606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:09.247824 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.247795    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.348305 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.348280    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.448753 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.448681    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.549182 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.549141    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.576330 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.576305    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 30 20:57:09.576935 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.576463    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 30 20:57:09.649825 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.649791    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.676166 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.676094    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 20:52:08 +0000 UTC" deadline="2028-02-17 10:02:06.466392247 +0000 UTC"
Jul 30 20:57:09.676166 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.676166    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13597h4m56.790235192s"
Jul 30 20:57:09.750832 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.750755    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.767611 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.767586    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 20:57:09.777094 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.777074    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 30 20:57:09.778095 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.778080    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientMemory"
Jul 30 20:57:09.778167 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.778115    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasNoDiskPressure"
Jul 30 20:57:09.778167 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.778129    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeHasSufficientPID"
Jul 30 20:57:09.789699 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:09.789659    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d952314a66eed41f4a061771aac559f.slice/crio-96442322f349484df036248bfd6afafc801080f3d5f2d7706f3baf1e31e7a53d WatchSource:0}: Error finding container 96442322f349484df036248bfd6afafc801080f3d5f2d7706f3baf1e31e7a53d: Status 404 returned error can't find the container with id 96442322f349484df036248bfd6afafc801080f3d5f2d7706f3baf1e31e7a53d
Jul 30 20:57:09.794481 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.794468    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 20:57:09.828416 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.828388    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 20:57:09.851602 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.851550    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.908479 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:09.908452    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba74c42e79d4ab3ec2af27545ae5c8d1.slice/crio-c139c027898c962e61f0ba6286e0e78fb1c8be8f15e658fa7c1b4dceab5c845e WatchSource:0}: Error finding container c139c027898c962e61f0ba6286e0e78fb1c8be8f15e658fa7c1b4dceab5c845e: Status 404 returned error can't find the container with id c139c027898c962e61f0ba6286e0e78fb1c8be8f15e658fa7c1b4dceab5c845e
Jul 30 20:57:09.952381 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:09.952356    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-139.ec2.internal\" not found"
Jul 30 20:57:09.977173 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:09.977152    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 20:57:10.043643 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.043554    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:10.056647 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.056626    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 20:57:10.057682 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.057667    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal"
Jul 30 20:57:10.065681 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.065668    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 30 20:57:10.624627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.624596    2583 apiserver.go:52] "Watching apiserver"
Jul 30 20:57:10.630384 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.630351    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 30 20:57:10.630887 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.630862    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-nlj6v","openshift-network-operator/iptables-alerter-b8zks","kube-system/global-pull-secret-syncer-gts47","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-9ftb6","openshift-multus/multus-r8txp","openshift-network-diagnostics/network-check-target-rxstk","openshift-ovn-kubernetes/ovnkube-node-h6n8x","kube-system/konnectivity-agent-sb66j","kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j","openshift-cluster-node-tuning-operator/tuned-r7h2z","openshift-dns/node-resolver-s2nhn","openshift-image-registry/node-ca-q5vql"]
Jul 30 20:57:10.632347 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.632320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.633756 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.633731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.634257 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.634238    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.634342 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.634283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qn2dq\""
Jul 30 20:57:10.634444 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.634427    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.634932 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.634914    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.635267 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635242    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-br6zp\""
Jul 30 20:57:10.635734 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 30 20:57:10.635734 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635618    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 30 20:57:10.635734 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635623    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.635924 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635793    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 30 20:57:10.635924 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.635811    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.636983 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.636545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.636983 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.636598    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 30 20:57:10.636983 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.636856    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 30 20:57:10.637217 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.637045    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 30 20:57:10.637217 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.637096    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.637457 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.637426    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.637558 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.637432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qxxj6\""
Jul 30 20:57:10.638230 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.638021    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 30 20:57:10.638230 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.638097    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 30 20:57:10.638362 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.638338    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7qb6j\""
Jul 30 20:57:10.638993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.638626    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 30 20:57:10.638993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.638672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.639521 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.639280    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.640282 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640066    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.640282 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640088    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.640413 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 30 20:57:10.640554 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640535    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8kpvk\""
Jul 30 20:57:10.640985 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.640985 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.640871    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4gggh\""
Jul 30 20:57:10.641381 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.641362    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.641466 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.641415    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.642071 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.642055    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.642162 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.642132    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:10.642476 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.642418    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.646655    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.646960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.647012    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.647177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.647415    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.647710    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.647800    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nn8jd\""
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.648329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:10.648441 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.648400    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:10.649898 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.649878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.650468 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.650450    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 30 20:57:10.650600 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.650583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k7l4d\""
Jul 30 20:57:10.651441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.651422    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 30 20:57:10.651585 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.651548    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 30 20:57:10.651808 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.651782    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 30 20:57:10.651901 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.651822    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h7524\""
Jul 30 20:57:10.655139 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:10.655232 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvsnh\" (UniqueName: \"kubernetes.io/projected/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kube-api-access-xvsnh\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.655232 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-netns\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655232 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-system-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.655232 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-hostroot\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-dbus\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-multus-certs\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-tmp-dir\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-systemd-units\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-systemd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655420    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysconfig\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knjcv\" (UniqueName: \"kubernetes.io/projected/fb77040d-4ca0-4065-a464-b926ff834073-kube-api-access-knjcv\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-registration-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f248\" (UniqueName: \"kubernetes.io/projected/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-kube-api-access-9f248\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-system-cni-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cnibin\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655591    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-netd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-lib-modules\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655636    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb77040d-4ca0-4065-a464-b926ff834073-host-slash\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-etc-kubernetes\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-bin\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-script-lib\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngvt\" (UniqueName: \"kubernetes.io/projected/de3a92fe-ffbc-4438-9801-81c920293a4f-kube-api-access-dngvt\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-sys-fs\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.655825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-k8s-cni-cncf-io\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-log-socket\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-env-overrides\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46b4b672-d044-4552-a556-d09e24baecd5-agent-certs\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-conf\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655944    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-sys\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-socket-dir-parent\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.655986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m46\" (UniqueName: \"kubernetes.io/projected/8982e187-c64c-44c8-8047-7c7b532aa65d-kube-api-access-r5m46\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-socket-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-daemon-config\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-slash\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-var-lib-kubelet\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.656533 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-device-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-kubelet-config\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-bin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-conf-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-hosts-file\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-kubelet\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-kubernetes\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb77040d-4ca0-4065-a464-b926ff834073-iptables-alerter-script\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-ovn\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3a92fe-ffbc-4438-9801-81c920293a4f-ovn-node-metrics-cert\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-tmp\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-run\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656504    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwxx\" (UniqueName: \"kubernetes.io/projected/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-kube-api-access-kwwxx\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-os-release\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-cni-binary-copy\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-var-lib-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-etc-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-host\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-cnibin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-node-log\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-modprobe-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5gm\" (UniqueName: \"kubernetes.io/projected/8a763470-0568-4720-a87c-636745df6053-kube-api-access-cr5gm\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.656984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-multus\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-kubelet\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-os-release\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwsc6\" (UniqueName: \"kubernetes.io/projected/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-kube-api-access-mwsc6\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46b4b672-d044-4552-a556-d09e24baecd5-konnectivity-ca\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-tuned\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.657993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-netns\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.658752 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.658752 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-config\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.658752 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.657248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-systemd\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.734030 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.734001    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbkx5"
Jul 30 20:57:10.743009 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.742983    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qbkx5"
Jul 30 20:57:10.745033 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.745012    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 30 20:57:10.757705 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-tmp-dir\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757710    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-systemd-units\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-systemd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysconfig\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knjcv\" (UniqueName: \"kubernetes.io/projected/fb77040d-4ca0-4065-a464-b926ff834073-kube-api-access-knjcv\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-registration-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f248\" (UniqueName: \"kubernetes.io/projected/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-kube-api-access-9f248\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-system-cni-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cnibin\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.757826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-netd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757852    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-lib-modules\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb77040d-4ca0-4065-a464-b926ff834073-host-slash\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-etc-kubernetes\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-bin\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-script-lib\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngvt\" (UniqueName: \"kubernetes.io/projected/de3a92fe-ffbc-4438-9801-81c920293a4f-kube-api-access-dngvt\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-sys-fs\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.757986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-k8s-cni-cncf-io\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-log-socket\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-tmp-dir\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-env-overrides\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46b4b672-d044-4552-a556-d09e24baecd5-agent-certs\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-conf\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-sys\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.758249 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-socket-dir-parent\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m46\" (UniqueName: \"kubernetes.io/projected/8982e187-c64c-44c8-8047-7c7b532aa65d-kube-api-access-r5m46\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-socket-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-systemd-units\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-systemd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-daemon-config\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysconfig\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpk4\" (UniqueName: \"kubernetes.io/projected/7e57f05e-01a1-43ac-abb3-32f6daed266e-kube-api-access-thpk4\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-slash\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-var-lib-kubelet\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-device-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-kubelet-config\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-bin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759017 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-conf-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-hosts-file\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-kubelet\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-kubernetes\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb77040d-4ca0-4065-a464-b926ff834073-iptables-alerter-script\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e57f05e-01a1-43ac-abb3-32f6daed266e-serviceca\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-ovn\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3a92fe-ffbc-4438-9801-81c920293a4f-ovn-node-metrics-cert\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758705    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-tmp\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e57f05e-01a1-43ac-abb3-32f6daed266e-host\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-run\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwxx\" (UniqueName: \"kubernetes.io/projected/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-kube-api-access-kwwxx\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-os-release\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-cni-binary-copy\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.759906 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-var-lib-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-var-lib-kubelet\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-etc-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-device-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-kubelet-config\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-bin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-conf-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-hosts-file\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-registration-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-socket-dir-parent\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cnibin\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-netd\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-conf\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.760842 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-env-overrides\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-lib-modules\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-run-ovn\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-sys\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-etc-kubernetes\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb77040d-4ca0-4065-a464-b926ff834073-iptables-alerter-script\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-cni-bin\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-system-cni-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-sysctl-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-kubelet\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-run\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.759706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb77040d-4ca0-4065-a464-b926ff834073-host-slash\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-socket-dir\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-os-release\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-script-lib\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-log-socket\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.761877 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-sys-fs\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-k8s-cni-cncf-io\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.758933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-etc-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-cni-binary-copy\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-host\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-cnibin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-cni-binary-copy\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-node-log\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-kubernetes\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-node-log\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-modprobe-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.760576    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5gm\" (UniqueName: \"kubernetes.io/projected/8a763470-0568-4720-a87c-636745df6053-kube-api-access-cr5gm\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-slash\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-multus\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.760645    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:11.260626009 +0000 UTC m=+3.026587873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-cni-multus\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.762799 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-cnibin\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-modprobe-d\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-var-lib-openvswitch\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-kubelet\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-os-release\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-host\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760803    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwsc6\" (UniqueName: \"kubernetes.io/projected/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-kube-api-access-mwsc6\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-var-lib-kubelet\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46b4b672-d044-4552-a556-d09e24baecd5-konnectivity-ca\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-tuned\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-netns\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-netns\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.760983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-os-release\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761320    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8982e187-c64c-44c8-8047-7c7b532aa65d-multus-daemon-config\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46b4b672-d044-4552-a556-d09e24baecd5-konnectivity-ca\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.763575 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761630    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-config\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-systemd\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvsnh\" (UniqueName: \"kubernetes.io/projected/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kube-api-access-xvsnh\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-netns\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-system-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-hostroot\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-dbus\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-multus-certs\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.761993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-host-run-multus-certs\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de3a92fe-ffbc-4438-9801-81c920293a4f-host-run-netns\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfa27313-a228-4ddc-abc6-78cb031f2476-dbus\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de3a92fe-ffbc-4438-9801-81c920293a4f-ovnkube-config\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.764297 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762446    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-systemd\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762498    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-system-cni-dir\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.762622    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8982e187-c64c-44c8-8047-7c7b532aa65d-hostroot\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.762879    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.762943    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:11.262927284 +0000 UTC m=+3.028889155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.764507    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de3a92fe-ffbc-4438-9801-81c920293a4f-ovn-node-metrics-cert\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.765101 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.764916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-tmp\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.765739 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.765717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-etc-tuned\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.768244 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.768220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46b4b672-d044-4552-a556-d09e24baecd5-agent-certs\") pod \"konnectivity-agent-sb66j\" (UID: \"46b4b672-d044-4552-a556-d09e24baecd5\") " pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.768908 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.768891    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:10.769000 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.768920    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:10.769000 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.768935    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:10.769105 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:10.769008    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:11.268991517 +0000 UTC m=+3.034953393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:10.774509 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.774482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwsc6\" (UniqueName: \"kubernetes.io/projected/ffa0da2f-098a-4e66-9d21-78c92c2b5b71-kube-api-access-mwsc6\") pod \"multus-additional-cni-plugins-9ftb6\" (UID: \"ffa0da2f-098a-4e66-9d21-78c92c2b5b71\") " pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.774670 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.774652    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvsnh\" (UniqueName: \"kubernetes.io/projected/db14f986-e3da-4886-a4c8-cbb6cd8a7a8f-kube-api-access-xvsnh\") pod \"aws-ebs-csi-driver-node-mdz6j\" (UID: \"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.774772 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.774748    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knjcv\" (UniqueName: \"kubernetes.io/projected/fb77040d-4ca0-4065-a464-b926ff834073-kube-api-access-knjcv\") pod \"iptables-alerter-b8zks\" (UID: \"fb77040d-4ca0-4065-a464-b926ff834073\") " pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.775018 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.774940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f248\" (UniqueName: \"kubernetes.io/projected/2a4ce903-51af-4a3b-a1be-ae00c1bccb0b-kube-api-access-9f248\") pod \"node-resolver-s2nhn\" (UID: \"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b\") " pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.777215 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.776380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m46\" (UniqueName: \"kubernetes.io/projected/8982e187-c64c-44c8-8047-7c7b532aa65d-kube-api-access-r5m46\") pod \"multus-r8txp\" (UID: \"8982e187-c64c-44c8-8047-7c7b532aa65d\") " pod="openshift-multus/multus-r8txp"
Jul 30 20:57:10.777215 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.776410    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwxx\" (UniqueName: \"kubernetes.io/projected/bfaebee1-ea4a-4394-ad0e-fdf2e093825e-kube-api-access-kwwxx\") pod \"tuned-r7h2z\" (UID: \"bfaebee1-ea4a-4394-ad0e-fdf2e093825e\") " pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.777365 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.777317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngvt\" (UniqueName: \"kubernetes.io/projected/de3a92fe-ffbc-4438-9801-81c920293a4f-kube-api-access-dngvt\") pod \"ovnkube-node-h6n8x\" (UID: \"de3a92fe-ffbc-4438-9801-81c920293a4f\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.777883 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.777857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5gm\" (UniqueName: \"kubernetes.io/projected/8a763470-0568-4720-a87c-636745df6053-kube-api-access-cr5gm\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:10.780838 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.780796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ba74c42e79d4ab3ec2af27545ae5c8d1","Type":"ContainerStarted","Data":"c139c027898c962e61f0ba6286e0e78fb1c8be8f15e658fa7c1b4dceab5c845e"}
Jul 30 20:57:10.783372 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.783351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"2d952314a66eed41f4a061771aac559f","Type":"ContainerStarted","Data":"96442322f349484df036248bfd6afafc801080f3d5f2d7706f3baf1e31e7a53d"}
Jul 30 20:57:10.862333 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.862286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpk4\" (UniqueName: \"kubernetes.io/projected/7e57f05e-01a1-43ac-abb3-32f6daed266e-kube-api-access-thpk4\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.862333 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.862338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e57f05e-01a1-43ac-abb3-32f6daed266e-serviceca\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.862587 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.862365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e57f05e-01a1-43ac-abb3-32f6daed266e-host\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.862587 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.862471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e57f05e-01a1-43ac-abb3-32f6daed266e-host\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.862938 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.862912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e57f05e-01a1-43ac-abb3-32f6daed266e-serviceca\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.876907 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.876768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpk4\" (UniqueName: \"kubernetes.io/projected/7e57f05e-01a1-43ac-abb3-32f6daed266e-kube-api-access-thpk4\") pod \"node-ca-q5vql\" (UID: \"7e57f05e-01a1-43ac-abb3-32f6daed266e\") " pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:10.947659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.947624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s2nhn"
Jul 30 20:57:10.954754 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.954728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9ftb6"
Jul 30 20:57:10.963063 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.963036    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:10.970051 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.969895    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:10.976674 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.976472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j"
Jul 30 20:57:10.983224 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.983199    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r7h2z"
Jul 30 20:57:10.990980 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.990952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8zks"
Jul 30 20:57:10.998661 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:10.998640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8txp"
Jul 30 20:57:11.005348 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.005331    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q5vql"
Jul 30 20:57:11.060431 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.060398    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 30 20:57:11.265385 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.265313    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:11.265385 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.265375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:11.265586 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.265470    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:11.265586 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.265481    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:11.265586 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.265537    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:12.265521681 +0000 UTC m=+4.031483538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:11.265586 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.265553    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:12.265546745 +0000 UTC m=+4.031508602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:11.366419 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.366374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:11.366637 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.366547    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:11.366637 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.366621    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:11.366637 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.366634    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:11.366802 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.366700    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:12.366681735 +0000 UTC m=+4.132643596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:11.745678 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.745559    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:52:10 +0000 UTC" deadline="2028-04-25 20:47:10.493412954 +0000 UTC"
Jul 30 20:57:11.745678 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.745628    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15239h49m58.747789475s"
Jul 30 20:57:11.771089 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:11.771056    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:11.771244 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:11.771159    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:11.905628 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.905600    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4ce903_51af_4a3b_a1be_ae00c1bccb0b.slice/crio-91522d86b51d128fa31f87bd56b29a40c8a52069c48c0961aad67c085a80447e WatchSource:0}: Error finding container 91522d86b51d128fa31f87bd56b29a40c8a52069c48c0961aad67c085a80447e: Status 404 returned error can't find the container with id 91522d86b51d128fa31f87bd56b29a40c8a52069c48c0961aad67c085a80447e
Jul 30 20:57:11.912422 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.911769    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb14f986_e3da_4886_a4c8_cbb6cd8a7a8f.slice/crio-4b759a32bd7a65e71e690377a5c48ef561aa33dd8ec09fc58baf33fcb42a2624 WatchSource:0}: Error finding container 4b759a32bd7a65e71e690377a5c48ef561aa33dd8ec09fc58baf33fcb42a2624: Status 404 returned error can't find the container with id 4b759a32bd7a65e71e690377a5c48ef561aa33dd8ec09fc58baf33fcb42a2624
Jul 30 20:57:11.912422 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.912378    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b4b672_d044_4552_a556_d09e24baecd5.slice/crio-05c1e9f21911fd776459a48ca75c3771b3ae9f8a0faa7dda231924a92ddf835f WatchSource:0}: Error finding container 05c1e9f21911fd776459a48ca75c3771b3ae9f8a0faa7dda231924a92ddf835f: Status 404 returned error can't find the container with id 05c1e9f21911fd776459a48ca75c3771b3ae9f8a0faa7dda231924a92ddf835f
Jul 30 20:57:11.913051 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.912974    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e57f05e_01a1_43ac_abb3_32f6daed266e.slice/crio-134cb046d5399adb7b35c99dcf6f792982c2052d80dba5da4a028501e05f1714 WatchSource:0}: Error finding container 134cb046d5399adb7b35c99dcf6f792982c2052d80dba5da4a028501e05f1714: Status 404 returned error can't find the container with id 134cb046d5399adb7b35c99dcf6f792982c2052d80dba5da4a028501e05f1714
Jul 30 20:57:11.914600 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.914581    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8982e187_c64c_44c8_8047_7c7b532aa65d.slice/crio-8717f954683fc3ee78632add76427c74cb0e2781adbab37868f15f0255e0f82f WatchSource:0}: Error finding container 8717f954683fc3ee78632add76427c74cb0e2781adbab37868f15f0255e0f82f: Status 404 returned error can't find the container with id 8717f954683fc3ee78632add76427c74cb0e2781adbab37868f15f0255e0f82f
Jul 30 20:57:11.915997 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.915396    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaebee1_ea4a_4394_ad0e_fdf2e093825e.slice/crio-4d35fd4ff769535410cc19e98e1cfcfbdfb7ac0e13e50789eb3e20e26e649bf4 WatchSource:0}: Error finding container 4d35fd4ff769535410cc19e98e1cfcfbdfb7ac0e13e50789eb3e20e26e649bf4: Status 404 returned error can't find the container with id 4d35fd4ff769535410cc19e98e1cfcfbdfb7ac0e13e50789eb3e20e26e649bf4
Jul 30 20:57:11.916860 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.916839    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa0da2f_098a_4e66_9d21_78c92c2b5b71.slice/crio-7622671eb28fab4c1d97770b7e6c8f69699a91638e4258e06e28277dedba12e6 WatchSource:0}: Error finding container 7622671eb28fab4c1d97770b7e6c8f69699a91638e4258e06e28277dedba12e6: Status 404 returned error can't find the container with id 7622671eb28fab4c1d97770b7e6c8f69699a91638e4258e06e28277dedba12e6
Jul 30 20:57:11.917131 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.917111    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3a92fe_ffbc_4438_9801_81c920293a4f.slice/crio-68983c5a1ed2a2f7ae20f3906d25f77ba876a95eca8d15ebe691b5e967868489 WatchSource:0}: Error finding container 68983c5a1ed2a2f7ae20f3906d25f77ba876a95eca8d15ebe691b5e967868489: Status 404 returned error can't find the container with id 68983c5a1ed2a2f7ae20f3906d25f77ba876a95eca8d15ebe691b5e967868489
Jul 30 20:57:11.917750 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:11.917730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb77040d_4ca0_4065_a464_b926ff834073.slice/crio-1b7a943ad3f4e8e26381bf3a704ad3ce66adc8e3f814c2b3ebc99d89cbe7e9da WatchSource:0}: Error finding container 1b7a943ad3f4e8e26381bf3a704ad3ce66adc8e3f814c2b3ebc99d89cbe7e9da: Status 404 returned error can't find the container with id 1b7a943ad3f4e8e26381bf3a704ad3ce66adc8e3f814c2b3ebc99d89cbe7e9da
Jul 30 20:57:12.273941 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.273739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:12.274096 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.273975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:12.274096 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.273912    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:12.274096 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.274057    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:14.274041598 +0000 UTC m=+6.040003455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:12.274096 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.274064    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:12.274238 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.274108    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:14.274093292 +0000 UTC m=+6.040055149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:12.374923 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.374887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:12.375118 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.375083    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:12.375118 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.375105    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:12.375118 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.375116    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:12.375269 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.375176    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:14.375157882 +0000 UTC m=+6.141119741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:12.746506 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.746416    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 20:52:10 +0000 UTC" deadline="2028-05-09 02:31:06.293370846 +0000 UTC"
Jul 30 20:57:12.746506 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.746459    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15557h33m53.546916862s"
Jul 30 20:57:12.771355 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.771324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:12.771515 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.771448    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:12.773812 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.773788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:12.773936 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:12.773914    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:12.798940 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.798869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s2nhn" event={"ID":"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b","Type":"ContainerStarted","Data":"91522d86b51d128fa31f87bd56b29a40c8a52069c48c0961aad67c085a80447e"}
Jul 30 20:57:12.808185 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.807944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" event={"ID":"2d952314a66eed41f4a061771aac559f","Type":"ContainerStarted","Data":"64884891d8c223f15fede5f01078121bc3649cd03b00450bb0e83ccb1d64a41b"}
Jul 30 20:57:12.815476 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.815384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8zks" event={"ID":"fb77040d-4ca0-4065-a464-b926ff834073","Type":"ContainerStarted","Data":"1b7a943ad3f4e8e26381bf3a704ad3ce66adc8e3f814c2b3ebc99d89cbe7e9da"}
Jul 30 20:57:12.818106 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.817912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"68983c5a1ed2a2f7ae20f3906d25f77ba876a95eca8d15ebe691b5e967868489"}
Jul 30 20:57:12.825820 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.825767    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-139.ec2.internal" podStartSLOduration=2.825741919 podStartE2EDuration="2.825741919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:12.824912597 +0000 UTC m=+4.590874474" watchObservedRunningTime="2026-07-30 20:57:12.825741919 +0000 UTC m=+4.591703808"
Jul 30 20:57:12.826346 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.826323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r7h2z" event={"ID":"bfaebee1-ea4a-4394-ad0e-fdf2e093825e","Type":"ContainerStarted","Data":"4d35fd4ff769535410cc19e98e1cfcfbdfb7ac0e13e50789eb3e20e26e649bf4"}
Jul 30 20:57:12.827800 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.827723    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sb66j" event={"ID":"46b4b672-d044-4552-a556-d09e24baecd5","Type":"ContainerStarted","Data":"05c1e9f21911fd776459a48ca75c3771b3ae9f8a0faa7dda231924a92ddf835f"}
Jul 30 20:57:12.829806 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.829658    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8txp" event={"ID":"8982e187-c64c-44c8-8047-7c7b532aa65d","Type":"ContainerStarted","Data":"8717f954683fc3ee78632add76427c74cb0e2781adbab37868f15f0255e0f82f"}
Jul 30 20:57:12.833074 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.833050    2583 generic.go:356] "Generic (PLEG): container finished" podID="ba74c42e79d4ab3ec2af27545ae5c8d1" containerID="767fa2cb3bca1cf8242b88cba330f80f0ebb9614e3748b47cde9cb6ac3b1b461" exitCode=0
Jul 30 20:57:12.833321 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.833300    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ba74c42e79d4ab3ec2af27545ae5c8d1","Type":"ContainerDied","Data":"767fa2cb3bca1cf8242b88cba330f80f0ebb9614e3748b47cde9cb6ac3b1b461"}
Jul 30 20:57:12.841151 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.841070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerStarted","Data":"7622671eb28fab4c1d97770b7e6c8f69699a91638e4258e06e28277dedba12e6"}
Jul 30 20:57:12.850805 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.850772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q5vql" event={"ID":"7e57f05e-01a1-43ac-abb3-32f6daed266e","Type":"ContainerStarted","Data":"134cb046d5399adb7b35c99dcf6f792982c2052d80dba5da4a028501e05f1714"}
Jul 30 20:57:12.863461 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:12.863409    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j" event={"ID":"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f","Type":"ContainerStarted","Data":"4b759a32bd7a65e71e690377a5c48ef561aa33dd8ec09fc58baf33fcb42a2624"}
Jul 30 20:57:13.771282 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:13.771233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:13.771801 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:13.771361    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:13.883597 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:13.883513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" event={"ID":"ba74c42e79d4ab3ec2af27545ae5c8d1","Type":"ContainerStarted","Data":"10bc26c78f81fe527c9f9769fb67507eebd7ce395e3a13a0be075a0dccb02ec2"}
Jul 30 20:57:14.292243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:14.292144    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:14.292243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:14.292218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:14.292502 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.292390    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:14.292502 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.292452    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:18.292433769 +0000 UTC m=+10.058395633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:14.292950 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.292865    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:14.292950 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.292911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:18.292895709 +0000 UTC m=+10.058857569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:14.392759 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:14.392718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:14.393413 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.392925    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:14.393413 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.392949    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:14.393413 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.392963    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:14.393413 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.393019    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:18.393001454 +0000 UTC m=+10.158963313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:14.774825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:14.774017    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:14.774825 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.774150    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:14.774825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:14.774244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:14.774825 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:14.774348    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:15.771625 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:15.771586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:15.771823 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:15.771715    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:16.773326 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:16.772679    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:16.773326 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:16.772808    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:16.773326 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:16.773156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:16.773326 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:16.773257    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:17.771351 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:17.771256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:17.771511 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:17.771383    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:18.328997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:18.329085    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.329209    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.329274    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:26.329255574 +0000 UTC m=+18.095217431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.329481    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:18.329607 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.329539    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:26.329521407 +0000 UTC m=+18.095483271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:18.430177 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:18.430143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:18.430368 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.430289    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:18.430368 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.430303    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:18.430368 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.430311    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:18.430557 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.430375    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:26.430358938 +0000 UTC m=+18.196320814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:18.772441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:18.772365    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:18.772605 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.772471    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:18.772605 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:18.772550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:18.772722 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:18.772671    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:19.771497 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:19.771017    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:19.771497 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:19.771149    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:20.771825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:20.771789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:20.772294 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:20.771929    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:20.772664 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:20.772643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:20.772846 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:20.772789    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:21.771493 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:21.771458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:21.771709 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:21.771593    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:22.771099 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:22.771018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:22.771532 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:22.771022    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:22.771532 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:22.771151    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:22.771532 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:22.771195    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:23.770961 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:23.770928    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:23.771117 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:23.771026    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:24.771215 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:24.770952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:24.771215 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:24.770998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:24.771215 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:24.771084    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:24.771215 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:24.771179    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:25.771315 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:25.771281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:25.771780 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:25.771394    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:26.390909 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:26.390873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:26.391093 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:26.390935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:26.391093 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.391052    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:26.391093 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.391064    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:26.391256 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.391127    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:42.391108525 +0000 UTC m=+34.157070385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:26.391256 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.391146    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:42.391137617 +0000 UTC m=+34.157099480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:26.492044 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:26.491965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:26.492386 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.492146    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:26.492386 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.492173    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:26.492386 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.492186    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:26.492386 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.492254    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:42.492236235 +0000 UTC m=+34.258198107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:26.771741 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:26.771649    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:26.772122 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.771792    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:26.772122 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:26.771969    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:26.772122 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:26.772067    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:27.771404 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:27.771367    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:27.771609 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:27.771480    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:28.772295 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:28.772254    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:28.772757 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:28.772366    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:28.772757 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:28.772422    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:28.772757 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:28.772539    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:29.771354 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:29.771321    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:29.771536 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:29.771430    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:30.772141 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.770968    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:30.772141 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:30.771279    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:30.772141 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.771034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:30.772141 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:30.771385    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:30.921879 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.921692    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 20:57:30.923148 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.923112    2583 generic.go:356] "Generic (PLEG): container finished" podID="de3a92fe-ffbc-4438-9801-81c920293a4f" containerID="d1742ff6e9670777399df6d3678e3e4145a4f329bcb117ffacd2685f7a7faf3e" exitCode=1
Jul 30 20:57:30.923257 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.923171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerDied","Data":"d1742ff6e9670777399df6d3678e3e4145a4f329bcb117ffacd2685f7a7faf3e"}
Jul 30 20:57:30.923364 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.923267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"1152d4679fc050d92d95e2948a9dc7847a11590dd3716519f2715bc11c9304af"}
Jul 30 20:57:30.925121 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.925101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r7h2z" event={"ID":"bfaebee1-ea4a-4394-ad0e-fdf2e093825e","Type":"ContainerStarted","Data":"877965c3ef815ad75b7ffaeb4aff792a6d5f17479157169fa9ba1ce507614d8f"}
Jul 30 20:57:30.926813 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.926793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sb66j" event={"ID":"46b4b672-d044-4552-a556-d09e24baecd5","Type":"ContainerStarted","Data":"da42df98aa4805135dea6cf1957124ae27953caf4e5d24ca9c30783c042d9b59"}
Jul 30 20:57:30.929214 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.929188    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8txp" event={"ID":"8982e187-c64c-44c8-8047-7c7b532aa65d","Type":"ContainerStarted","Data":"f367cad71376f35032251c9a80331385637a5b0caa98275f62c4997c71719890"}
Jul 30 20:57:30.930854 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.930830    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerStarted","Data":"1ea1d8a50e6056d79fc9c2b6a0d1ae33bf079cfc7b5f5a8bdc36a48466a7d6ea"}
Jul 30 20:57:30.932338 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.932310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q5vql" event={"ID":"7e57f05e-01a1-43ac-abb3-32f6daed266e","Type":"ContainerStarted","Data":"b08b05a1a02b56d1e6aaa39670dc13b882f9e96265211f2ab8bde0b8a43157dd"}
Jul 30 20:57:30.933908 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.933887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j" event={"ID":"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f","Type":"ContainerStarted","Data":"1a59815bfa787d7b2865db45b5ca91da7cd473f5c3a96eefa795d06f87c02700"}
Jul 30 20:57:30.935371 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.935350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s2nhn" event={"ID":"2a4ce903-51af-4a3b-a1be-ae00c1bccb0b","Type":"ContainerStarted","Data":"61fc64dea347c207f983379b5b118dd756e2ce0608279f0d8fb47ce104f67a19"}
Jul 30 20:57:30.940920 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.940884    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r7h2z" podStartSLOduration=4.310042524 podStartE2EDuration="22.940872002s" totalImagesPullingTime="18.630829478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:30.940479331 +0000 UTC m=+22.706441212" watchObservedRunningTime="2026-07-30 20:57:30.940872002 +0000 UTC m=+22.706833880"
Jul 30 20:57:30.941128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.941108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-139.ec2.internal" podStartSLOduration=20.941102926 podStartE2EDuration="20.941102926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:13.898773839 +0000 UTC m=+5.664735717" watchObservedRunningTime="2026-07-30 20:57:30.941102926 +0000 UTC m=+22.707064805"
Jul 30 20:57:30.956317 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.956260    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r8txp" podStartSLOduration=3.289608535 podStartE2EDuration="21.956239255s" totalImagesPullingTime="18.66663072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:30.955807742 +0000 UTC m=+22.721769620" watchObservedRunningTime="2026-07-30 20:57:30.956239255 +0000 UTC m=+22.722201167"
Jul 30 20:57:30.979152 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:30.979108    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q5vql" podStartSLOduration=3.344646697 podStartE2EDuration="21.979093223s" totalImagesPullingTime="18.634446526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:30.979080004 +0000 UTC m=+22.745041883" watchObservedRunningTime="2026-07-30 20:57:30.979093223 +0000 UTC m=+22.745055101"
Jul 30 20:57:31.031286 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.031071    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s2nhn" podStartSLOduration=4.421759173 podStartE2EDuration="23.031052804s" totalImagesPullingTime="18.609293631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:31.004786986 +0000 UTC m=+22.770748864" watchObservedRunningTime="2026-07-30 20:57:31.031052804 +0000 UTC m=+22.797014687"
Jul 30 20:57:31.770942 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.770913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:31.771179 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:31.771028    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:31.938879 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.938843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8zks" event={"ID":"fb77040d-4ca0-4065-a464-b926ff834073","Type":"ContainerStarted","Data":"b6ad2690cefd3d64daf8b332e981dfc7629b66a21b23457ce4ad92be4453cd61"}
Jul 30 20:57:31.942688 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.942666    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 20:57:31.943133 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.943098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"9219c9367a9e98558cde8d5aa9c41b5bbd29fd3f67b3afcc3417293bf64219df"}
Jul 30 20:57:31.943248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.943180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"43de0212bacbdd542e0011b46ac8214ab9cca499a41b33d9f3ecb9f5befeca1d"}
Jul 30 20:57:31.943248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.943226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"624d27a87fa2e29e48309fca34eb23e095680cfb51da319d84bb628e5ef23070"}
Jul 30 20:57:31.943331 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.943292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"67bf9f36979c896cb878e38a1120bd520b4eda29228edd5b58eab69352743594"}
Jul 30 20:57:31.944812 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.944788    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="1ea1d8a50e6056d79fc9c2b6a0d1ae33bf079cfc7b5f5a8bdc36a48466a7d6ea" exitCode=0
Jul 30 20:57:31.944908 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.944882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"1ea1d8a50e6056d79fc9c2b6a0d1ae33bf079cfc7b5f5a8bdc36a48466a7d6ea"}
Jul 30 20:57:31.951021 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.950985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b8zks" podStartSLOduration=4.353608569 podStartE2EDuration="22.950971605s" totalImagesPullingTime="18.597363036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:31.950872836 +0000 UTC m=+23.716834715" watchObservedRunningTime="2026-07-30 20:57:31.950971605 +0000 UTC m=+23.716933483"
Jul 30 20:57:31.951458 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:31.951431    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sb66j" podStartSLOduration=9.629175364 podStartE2EDuration="23.951420523s" totalImagesPullingTime="14.322245159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:31.044274769 +0000 UTC m=+22.810236627" watchObservedRunningTime="2026-07-30 20:57:31.951420523 +0000 UTC m=+23.717382393"
Jul 30 20:57:32.472903 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.472730    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 30 20:57:32.712895 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.712720    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T20:57:32.472900605Z","UUID":"655990f8-4110-470e-952a-703fb3300967","Handler":null,"Name":"","Endpoint":""}
Jul 30 20:57:32.714126 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.714107    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 30 20:57:32.714126 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.714140    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 30 20:57:32.771640 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.771608    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:32.771640 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.771631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:32.771867 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:32.771724    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:32.771931 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:32.771886    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:32.949536 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:32.949490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j" event={"ID":"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f","Type":"ContainerStarted","Data":"c52a3250b7d5c3d95a5f90d207610f2ce649be6e5d671e439b94d2ec784eedd6"}
Jul 30 20:57:33.770949 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:33.770876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:33.771089 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:33.770994    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:33.954840 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:33.954808    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 20:57:33.955361 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:33.955183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"7ec53cf1fe69a5864d7d9bd67cec872fcb7ce3eb5fc14512fb44238e0efb9831"}
Jul 30 20:57:33.957176 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:33.957152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j" event={"ID":"db14f986-e3da-4886-a4c8-cbb6cd8a7a8f","Type":"ContainerStarted","Data":"3b6876e0aac5287d49dda84d9a913ef427fa29c82afaa3818c6234f4c611cda7"}
Jul 30 20:57:33.974393 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:33.974352    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mdz6j" podStartSLOduration=4.8618188799999995 podStartE2EDuration="25.97433964s" totalImagesPullingTime="21.11252076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:33.974204135 +0000 UTC m=+25.740166013" watchObservedRunningTime="2026-07-30 20:57:33.97433964 +0000 UTC m=+25.740301518"
Jul 30 20:57:34.448081 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:34.448039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:34.448714 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:34.448684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:34.770890 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:34.770800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:34.770890 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:34.770820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:34.771088 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:34.770929    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:34.771088 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:34.771030    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:35.771609 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:35.771561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:35.772101 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:35.771691    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:36.252266 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:36.252221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:36.252466 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:36.252362    2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 30 20:57:36.252884 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:36.252837    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sb66j"
Jul 30 20:57:36.772028 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:36.771750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:36.772028 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:36.771759    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:36.772028 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:36.771906    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:36.772028 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:36.771975    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:37.771910 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.771729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:37.772082 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:37.771974    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:37.968727 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.968692    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="b43a7e25021a55a32db532ebbe6dc8f5ce898bd0ae110b6943d2f45df1db1f5a" exitCode=0
Jul 30 20:57:37.968878 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.968779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"b43a7e25021a55a32db532ebbe6dc8f5ce898bd0ae110b6943d2f45df1db1f5a"}
Jul 30 20:57:37.972216 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.972194    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 20:57:37.972551 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.972533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"dd1de0cda9472f0e5c723073fdc841335403f688eab72f02aff88beea6b34694"}
Jul 30 20:57:37.972803 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.972787    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:37.972862 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.972852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:37.972920 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.972913    2583 scope.go:117] "RemoveContainer" containerID="d1742ff6e9670777399df6d3678e3e4145a4f329bcb117ffacd2685f7a7faf3e"
Jul 30 20:57:37.990579 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:37.990530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:38.772463 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.772141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:38.772463 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.772141    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:38.773148 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:38.772588    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:38.773148 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:38.772773    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:38.979244 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.979222    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 20:57:38.979529 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.979509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" event={"ID":"de3a92fe-ffbc-4438-9801-81c920293a4f","Type":"ContainerStarted","Data":"3b2a3295296db0b29258db8fd59534758e04080b977b8deb7d7a43c922c9d94c"}
Jul 30 20:57:38.979748 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.979727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:38.994816 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:38.994795    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:57:39.008025 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.007985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x" podStartSLOduration=12.314208153 podStartE2EDuration="31.007970469s" totalImagesPullingTime="18.693762316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:39.006884356 +0000 UTC m=+30.772846237" watchObservedRunningTime="2026-07-30 20:57:39.007970469 +0000 UTC m=+30.773932348"
Jul 30 20:57:39.057523 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.057482    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gts47"]
Jul 30 20:57:39.057721 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.057621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:39.057796 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:39.057771    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:39.060376 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.060352    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlj6v"]
Jul 30 20:57:39.060496 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.060458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:39.060606 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:39.060583    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:39.063582 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.063541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxstk"]
Jul 30 20:57:39.063691 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.063658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:39.063781 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:39.063760    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:39.984303 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.984216    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="f67adf2b20e4ec03b453eefc3aed8a10dbd919a163c3f7ff8feafd5667a2a32c" exitCode=0
Jul 30 20:57:39.984869 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:39.984293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"f67adf2b20e4ec03b453eefc3aed8a10dbd919a163c3f7ff8feafd5667a2a32c"}
Jul 30 20:57:40.771544 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:40.771420    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:40.771544 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:40.771452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:40.771669 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:40.771540    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:40.771669 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:40.771607    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:40.771669 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:40.771665    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:40.771772 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:40.771730    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:40.988921 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:40.988890    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="81569131961a998130607c82cd0407ad27588b633baba64b6c99d42b45a1aaad" exitCode=0
Jul 30 20:57:40.989387 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:40.988975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"81569131961a998130607c82cd0407ad27588b633baba64b6c99d42b45a1aaad"}
Jul 30 20:57:42.412684 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.412594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:42.412684 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.412659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:42.413192 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.412751    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:42.413192 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.412759    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:42.413192 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.412808    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret podName:dfa27313-a228-4ddc-abc6-78cb031f2476 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:14.412796463 +0000 UTC m=+66.178758319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret") pod "global-pull-secret-syncer-gts47" (UID: "dfa27313-a228-4ddc-abc6-78cb031f2476") : object "kube-system"/"original-pull-secret" not registered
Jul 30 20:57:42.413192 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.412823    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:14.412816512 +0000 UTC m=+66.178778369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 30 20:57:42.513503 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.513467    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:42.513704 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.513676    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 30 20:57:42.513774 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.513707    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 30 20:57:42.513774 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.513720    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bszj4 for pod openshift-network-diagnostics/network-check-target-rxstk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:42.513860 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.513788    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4 podName:e0f6e363-dd36-41f1-8089-e003ab134549 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:14.513769454 +0000 UTC m=+66.279731324 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bszj4" (UniqueName: "kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4") pod "network-check-target-rxstk" (UID: "e0f6e363-dd36-41f1-8089-e003ab134549") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 30 20:57:42.775850 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.775776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:42.776019 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.775777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:42.776019 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.775880    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rxstk" podUID="e0f6e363-dd36-41f1-8089-e003ab134549"
Jul 30 20:57:42.776019 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:42.775780    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:42.776019 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.775965    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nlj6v" podUID="8a763470-0568-4720-a87c-636745df6053"
Jul 30 20:57:42.776226 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:42.776068    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gts47" podUID="dfa27313-a228-4ddc-abc6-78cb031f2476"
Jul 30 20:57:43.544035 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.543999    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-139.ec2.internal" event="NodeReady"
Jul 30 20:57:43.544520 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.544154    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 30 20:57:43.576268 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.576229    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:57:43.578165 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.578142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.579455 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.579427    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"]
Jul 30 20:57:43.580617 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.580599    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 30 20:57:43.581150 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.581130    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-klzqk\""
Jul 30 20:57:43.581369 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.581351    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"]
Jul 30 20:57:43.581494 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.581479    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.582132 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.582115    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 30 20:57:43.583248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.583226    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 30 20:57:43.583992 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.583974    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"]
Jul 30 20:57:43.584107 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.584096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.585577 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.585539    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 30 20:57:43.585800 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.585784    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.586437 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586281    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 30 20:57:43.586437 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 30 20:57:43.586609 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 30 20:57:43.586609 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586538    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 30 20:57:43.586792 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 30 20:57:43.586853 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 30 20:57:43.586977 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.586960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 30 20:57:43.587675 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.587394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 30 20:57:43.588038 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.587757    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-kk6tm\""
Jul 30 20:57:43.590666 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.590394    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 30 20:57:43.595248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.595177    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:57:43.599536 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.599513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"]
Jul 30 20:57:43.599662 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.599553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"]
Jul 30 20:57:43.603145 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.603122    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"]
Jul 30 20:57:43.603851 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.603815    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-flp9x"]
Jul 30 20:57:43.606131 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.606108    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zl2hb"]
Jul 30 20:57:43.606271 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.606252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.608349 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.608329    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.608471 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.608448    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xbdm8\""
Jul 30 20:57:43.608821 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.608804    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 30 20:57:43.609066 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.609025    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 30 20:57:43.609132 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.609082    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 30 20:57:43.610644 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.610623    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 30 20:57:43.610825 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.610810    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 30 20:57:43.610878 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.610848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-77cz9\""
Jul 30 20:57:43.613147 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.613110    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-flp9x"]
Jul 30 20:57:43.617481 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.617458    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zl2hb"]
Jul 30 20:57:43.724311 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fss\" (UniqueName: \"kubernetes.io/projected/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-kube-api-access-22fss\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.724492 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.724492 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724356    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.724492 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b117ac-ff6d-4f65-b77c-4f35f994a462-tmp\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.724492 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrxq\" (UniqueName: \"kubernetes.io/projected/98d00d12-ce16-4264-91a2-eb4edb987364-kube-api-access-mlrxq\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwst\" (UniqueName: \"kubernetes.io/projected/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-kube-api-access-slwst\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724658    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.724718 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-tmp-dir\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmbs\" (UniqueName: \"kubernetes.io/projected/8e791680-1811-4ace-a28c-49f1ebd6b997-kube-api-access-sbmbs\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-cert\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724868    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d00d12-ce16-4264-91a2-eb4edb987364-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.724969 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srvph\" (UniqueName: \"kubernetes.io/projected/d3b117ac-ff6d-4f65-b77c-4f35f994a462-kube-api-access-srvph\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e791680-1811-4ace-a28c-49f1ebd6b997-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.724997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-config-volume\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.725019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d3b117ac-ff6d-4f65-b77c-4f35f994a462-klusterlet-config\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.725046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.725076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxpj\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.725095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.725240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.725125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.779097 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.779060    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:43.826066 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.825984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826066 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fss\" (UniqueName: \"kubernetes.io/projected/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-kube-api-access-22fss\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.826066 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826071    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826100    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b117ac-ff6d-4f65-b77c-4f35f994a462-tmp\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrxq\" (UniqueName: \"kubernetes.io/projected/98d00d12-ce16-4264-91a2-eb4edb987364-kube-api-access-mlrxq\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwst\" (UniqueName: \"kubernetes.io/projected/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-kube-api-access-slwst\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826194    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-tmp-dir\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.826309 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmbs\" (UniqueName: \"kubernetes.io/projected/8e791680-1811-4ace-a28c-49f1ebd6b997-kube-api-access-sbmbs\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-cert\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d00d12-ce16-4264-91a2-eb4edb987364-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srvph\" (UniqueName: \"kubernetes.io/projected/d3b117ac-ff6d-4f65-b77c-4f35f994a462-kube-api-access-srvph\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e791680-1811-4ace-a28c-49f1ebd6b997-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826530    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-config-volume\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d3b117ac-ff6d-4f65-b77c-4f35f994a462-klusterlet-config\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxpj\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.826659 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.826649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.827135 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:43.826791    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:43.827135 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:43.826852    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:44.326834111 +0000 UTC m=+36.092795973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:43.827210 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.827132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.827498 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:43.827473    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:43.827498 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:43.827495    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:43.827664 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:43.827540    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:57:44.327523849 +0000 UTC m=+36.093485722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:57:43.827838 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.827819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.828791 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.827925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e791680-1811-4ace-a28c-49f1ebd6b997-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.828791 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.828115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3b117ac-ff6d-4f65-b77c-4f35f994a462-tmp\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.828791 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.828418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-tmp-dir\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.829162 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.829133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.830008 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.829946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-config-volume\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.833152 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.832984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.833243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.832997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.833243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.833002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-cert\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.833243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.833002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.833243 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.833054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-ca\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.833695 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.833670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.834725 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.834705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d3b117ac-ff6d-4f65-b77c-4f35f994a462-klusterlet-config\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.834974 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.834950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/98d00d12-ce16-4264-91a2-eb4edb987364-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.838204 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.838163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwst\" (UniqueName: \"kubernetes.io/projected/edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2-kube-api-access-slwst\") pod \"ingress-canary-flp9x\" (UID: \"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2\") " pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:43.838811 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.838789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrxq\" (UniqueName: \"kubernetes.io/projected/98d00d12-ce16-4264-91a2-eb4edb987364-kube-api-access-mlrxq\") pod \"managed-serviceaccount-addon-agent-5fff4f46d8-562g6\" (UID: \"98d00d12-ce16-4264-91a2-eb4edb987364\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.838922 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.838846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmbs\" (UniqueName: \"kubernetes.io/projected/8e791680-1811-4ace-a28c-49f1ebd6b997-kube-api-access-sbmbs\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.838983 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.838955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fss\" (UniqueName: \"kubernetes.io/projected/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-kube-api-access-22fss\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:43.840178 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.840149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.840697 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.840675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srvph\" (UniqueName: \"kubernetes.io/projected/d3b117ac-ff6d-4f65-b77c-4f35f994a462-kube-api-access-srvph\") pod \"klusterlet-addon-workmgr-597596f55-k65kz\" (UID: \"d3b117ac-ff6d-4f65-b77c-4f35f994a462\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.840791 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.840752    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxpj\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:43.841042 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.841020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8e791680-1811-4ace-a28c-49f1ebd6b997-hub\") pod \"cluster-proxy-proxy-agent-6549dc97ff-rcrp5\" (UID: \"8e791680-1811-4ace-a28c-49f1ebd6b997\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.866102 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.866076    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:43.908210 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.908177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:57:43.918995 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.918976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:43.925735 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.925713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"
Jul 30 20:57:43.951236 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:43.951188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-flp9x"
Jul 30 20:57:44.098678 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.098356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"]
Jul 30 20:57:44.103072 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:44.103041    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e791680_1811_4ace_a28c_49f1ebd6b997.slice/crio-70e6c58ed773559c685423b1a29cc9414ec3aa59b2653d068462af877a811a7d WatchSource:0}: Error finding container 70e6c58ed773559c685423b1a29cc9414ec3aa59b2653d068462af877a811a7d: Status 404 returned error can't find the container with id 70e6c58ed773559c685423b1a29cc9414ec3aa59b2653d068462af877a811a7d
Jul 30 20:57:44.103483 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.103463    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"]
Jul 30 20:57:44.107463 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.107385    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6"]
Jul 30 20:57:44.108879 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:44.108849    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b117ac_ff6d_4f65_b77c_4f35f994a462.slice/crio-d92df92e3306f895ecabc5fe91d7d9285cb9c955635efb0004b834a40f37f7e2 WatchSource:0}: Error finding container d92df92e3306f895ecabc5fe91d7d9285cb9c955635efb0004b834a40f37f7e2: Status 404 returned error can't find the container with id d92df92e3306f895ecabc5fe91d7d9285cb9c955635efb0004b834a40f37f7e2
Jul 30 20:57:44.111443 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:44.111423    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98d00d12_ce16_4264_91a2_eb4edb987364.slice/crio-ab363babb4662fba1e49e7d34b72ac53db6cfd5e7c0e0d2d3a40dc41103c1055 WatchSource:0}: Error finding container ab363babb4662fba1e49e7d34b72ac53db6cfd5e7c0e0d2d3a40dc41103c1055: Status 404 returned error can't find the container with id ab363babb4662fba1e49e7d34b72ac53db6cfd5e7c0e0d2d3a40dc41103c1055
Jul 30 20:57:44.130660 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.130636    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-flp9x"]
Jul 30 20:57:44.330241 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.330209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:44.330426 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:44.330403    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:44.330426 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:44.330424    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:44.330601 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:44.330478    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:57:45.330463624 +0000 UTC m=+37.096425481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:57:44.330844 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.330823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:44.330956 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:44.330944    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:44.331007 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:44.330991    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:45.330975598 +0000 UTC m=+37.096937456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:44.603007 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.602969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:44.682250 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.682213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:44.775248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.775222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:57:44.775248 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.775243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:57:44.775478 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.775393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:57:44.777948 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.777920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 20:57:44.777948 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.777920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 20:57:44.778136 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.777957    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vkk\""
Jul 30 20:57:44.778136 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.777972    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xh5pq\""
Jul 30 20:57:44.778136 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.777975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 20:57:44.778136 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.778067    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 20:57:44.782220 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:44.782193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:45.003838 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.003800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6" event={"ID":"98d00d12-ce16-4264-91a2-eb4edb987364","Type":"ContainerStarted","Data":"ab363babb4662fba1e49e7d34b72ac53db6cfd5e7c0e0d2d3a40dc41103c1055"}
Jul 30 20:57:45.005022 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.004999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz" event={"ID":"d3b117ac-ff6d-4f65-b77c-4f35f994a462","Type":"ContainerStarted","Data":"d92df92e3306f895ecabc5fe91d7d9285cb9c955635efb0004b834a40f37f7e2"}
Jul 30 20:57:45.006134 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.006107    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerStarted","Data":"70e6c58ed773559c685423b1a29cc9414ec3aa59b2653d068462af877a811a7d"}
Jul 30 20:57:45.298496 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.298406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:45.338545 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.338503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:45.338755 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.338583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:45.338755 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:45.338694    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:45.338755 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:45.338699    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:45.338755 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:45.338725    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:45.338936 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:45.338776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:47.338757666 +0000 UTC m=+39.104719525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:45.338936 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:45.338797    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:57:47.338788138 +0000 UTC m=+39.104749995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:57:45.403029 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.402987    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:45.660028 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.659962    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:45.668431 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:57:45.668402    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedaefa0a_8e43_40cd_a68b_1bf0a3ee60e2.slice/crio-341b7406e9d45c6d92d63654a702557e283bbafe7b0c2940173784846aaf8eca WatchSource:0}: Error finding container 341b7406e9d45c6d92d63654a702557e283bbafe7b0c2940173784846aaf8eca: Status 404 returned error can't find the container with id 341b7406e9d45c6d92d63654a702557e283bbafe7b0c2940173784846aaf8eca
Jul 30 20:57:45.760470 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:45.760437    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:46.009977 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:46.009941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-flp9x" event={"ID":"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2","Type":"ContainerStarted","Data":"341b7406e9d45c6d92d63654a702557e283bbafe7b0c2940173784846aaf8eca"}
Jul 30 20:57:46.614473 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:46.614187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:47.020025 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:47.019989    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="b38896d00463120caf1ea595d005ae138f1989243fbbf4bd9101a406819321a1" exitCode=0
Jul 30 20:57:47.020630 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:47.020071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"b38896d00463120caf1ea595d005ae138f1989243fbbf4bd9101a406819321a1"}
Jul 30 20:57:47.356762 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:47.356726    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:47.356921 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:47.356791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:47.356999 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:47.356950    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:47.356999 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:47.356965    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:47.357113 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:47.357020    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:57:51.357002728 +0000 UTC m=+43.122964588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:57:47.357515 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:47.357490    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:47.357613 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:47.357551    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:51.35753483 +0000 UTC m=+43.123496695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:48.029959 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:48.029920    2583 generic.go:356] "Generic (PLEG): container finished" podID="ffa0da2f-098a-4e66-9d21-78c92c2b5b71" containerID="a3b50024ffb739e7df14accfb42c6aff19341d539492ace4ffe2bfcab770285b" exitCode=0
Jul 30 20:57:48.030486 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:48.030004    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerDied","Data":"a3b50024ffb739e7df14accfb42c6aff19341d539492ace4ffe2bfcab770285b"}
Jul 30 20:57:48.938832 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:48.938755    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:49.233171 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:49.233140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:49.332846 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:49.332813    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:49.433436 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:49.433406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:51.394443 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:51.394398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:51.394465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:51.394581    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:51.394640    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:51.394654    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:51.394662    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:57:59.3946381 +0000 UTC m=+51.160599965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:51.394903 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:51.394705    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:57:59.394689111 +0000 UTC m=+51.160650968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:57:51.840901 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:51.840842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:51.902912 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:51.902880    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:55.054684 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.054647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" event={"ID":"ffa0da2f-098a-4e66-9d21-78c92c2b5b71","Type":"ContainerStarted","Data":"f2b684b82237e0b3026e64033f7bf691d60a3f8ddff0fd6a308f86a42635813f"}
Jul 30 20:57:55.056086 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.056063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6" event={"ID":"98d00d12-ce16-4264-91a2-eb4edb987364","Type":"ContainerStarted","Data":"d6d1ff35e5fdfd5858affb961c096ca0bd5aa74b019dab4392691cd6610e74b4"}
Jul 30 20:57:55.061164 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.061133    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz" event={"ID":"d3b117ac-ff6d-4f65-b77c-4f35f994a462","Type":"ContainerStarted","Data":"5eb271f0ed26a7d49b8f409862482f74404fd2a4711d054996b8ef1ca084569e"}
Jul 30 20:57:55.061277 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.061249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:55.063282 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.063263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz"
Jul 30 20:57:55.063664 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.063639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerStarted","Data":"aa607eef0d2181b23f3864c0650c12a0c36e69ec0913d5842f6fc4a8e499edde"}
Jul 30 20:57:55.063785 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.063671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerStarted","Data":"3666458758a3e8d7884900f8b1892e3e8c066480a8522e81475b3ca81f7ab6fd"}
Jul 30 20:57:55.063785 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.063687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerStarted","Data":"9e05813408e0782f060a4e48f7111233f3f357aab912b1a754ef30405fa24df0"}
Jul 30 20:57:55.064977 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.064957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-flp9x" event={"ID":"edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2","Type":"ContainerStarted","Data":"103182d61d8a4ae1d4ecf7614b8ab7f9d6d29bc23a3fcaba36dd3d203d4318d1"}
Jul 30 20:57:55.077159 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.077116    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9ftb6" podStartSLOduration=16.535057075 podStartE2EDuration="47.07710407s" totalImagesPullingTime="30.542046995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:55.076308732 +0000 UTC m=+46.842270611" watchObservedRunningTime="2026-07-30 20:57:55.07710407 +0000 UTC m=+46.843065928"
Jul 30 20:57:55.090584 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.090518    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-flp9x" podStartSLOduration=3.491492776 podStartE2EDuration="12.090482507s" totalImagesPullingTime="8.598989731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:55.090088037 +0000 UTC m=+46.856049915" watchObservedRunningTime="2026-07-30 20:57:55.090482507 +0000 UTC m=+46.856444365"
Jul 30 20:57:55.104117 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.104055    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-597596f55-k65kz" podStartSLOduration=22.94625777 podStartE2EDuration="33.104040238s" totalImagesPullingTime="10.157782468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:55.103696274 +0000 UTC m=+46.869658154" watchObservedRunningTime="2026-07-30 20:57:55.104040238 +0000 UTC m=+46.870002095"
Jul 30 20:57:55.116803 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.116761    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5fff4f46d8-562g6" podStartSLOduration=22.962668217 podStartE2EDuration="33.116747309s" totalImagesPullingTime="10.154079092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:55.116323195 +0000 UTC m=+46.882285074" watchObservedRunningTime="2026-07-30 20:57:55.116747309 +0000 UTC m=+46.882709188"
Jul 30 20:57:55.133067 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.132987    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" podStartSLOduration=22.954682909 podStartE2EDuration="33.132967629s" totalImagesPullingTime="10.17828472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:57:55.132380196 +0000 UTC m=+46.898342075" watchObservedRunningTime="2026-07-30 20:57:55.132967629 +0000 UTC m=+46.898929509"
Jul 30 20:57:55.206836 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.206805    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:55.262582 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:55.262530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:57.350260 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:57.350225    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:57.409284 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:57.409258    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:57:59.455003 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:59.454962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:57:59.455003 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:57:59.455017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:57:59.455459 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:59.455133    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:57:59.455459 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:59.455134    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:57:59.455459 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:59.455204    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:15.455187976 +0000 UTC m=+67.221149834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:57:59.455459 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:59.455144    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:57:59.455459 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:57:59.455287    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:58:15.455274904 +0000 UTC m=+67.221236761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:58:00.713713 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:00.713672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:00.813550 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:00.813513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:00.866506 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:00.866478    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:05.883683 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:05.883647    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:05.984973 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:05.984945    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:08.741986 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:08.741946    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:08.800451 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:08.800421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:10.218407 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.218376    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:10.319089 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.319051    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:10.805036 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.805005    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f9glr"]
Jul 30 20:58:10.807767 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.807745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.809553 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.809528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 30 20:58:10.809872 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.809854    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ws5tw\""
Jul 30 20:58:10.809999 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.809969    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 30 20:58:10.810111 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.809974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 30 20:58:10.810111 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.809974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 30 20:58:10.821196 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.821174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f9glr"]
Jul 30 20:58:10.824594 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.824548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:10.838672 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.838647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.838783 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.838737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t67r\" (UniqueName: \"kubernetes.io/projected/dcf45e82-3aab-44db-a754-efe881059ce7-kube-api-access-5t67r\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.838826 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.838792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.838864 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.838823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcf45e82-3aab-44db-a754-efe881059ce7-data-volume\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.838927 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.838912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dcf45e82-3aab-44db-a754-efe881059ce7-crio-socket\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.883972 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.883941    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:10.940271 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940204    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dcf45e82-3aab-44db-a754-efe881059ce7-crio-socket\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t67r\" (UniqueName: \"kubernetes.io/projected/dcf45e82-3aab-44db-a754-efe881059ce7-kube-api-access-5t67r\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dcf45e82-3aab-44db-a754-efe881059ce7-crio-socket\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940609 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcf45e82-3aab-44db-a754-efe881059ce7-data-volume\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.940609 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:10.940435    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:10.940609 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:10.940497    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 20:58:10.940609 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:10.940551    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:11.440528198 +0000 UTC m=+63.206490067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:10.940609 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:10.940596    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:11.440583743 +0000 UTC m=+63.206545615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : configmap "kube-rbac-proxy" not found
Jul 30 20:58:10.940786 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.940695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcf45e82-3aab-44db-a754-efe881059ce7-data-volume\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:10.949230 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:10.949205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t67r\" (UniqueName: \"kubernetes.io/projected/dcf45e82-3aab-44db-a754-efe881059ce7-kube-api-access-5t67r\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:11.000090 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.000068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6n8x"
Jul 30 20:58:11.171090 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.171010    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:11.270377 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.270343    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:11.444793 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.444698    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:11.444951 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.444796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:11.444951 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:11.444862    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:11.444951 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:11.444871    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 20:58:11.444951 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:11.444940    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:12.444919545 +0000 UTC m=+64.210881402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:11.444951 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:11.444954    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:12.444947866 +0000 UTC m=+64.210909723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : configmap "kube-rbac-proxy" not found
Jul 30 20:58:11.702990 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:11.702913    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.129319 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.129285    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.240962 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.240925    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.322532 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.322499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.423710 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.423623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.452884 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.452853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:12.453014 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.452919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:12.453014 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:12.452988    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 20:58:12.453112 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:12.453020    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:12.453112 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:12.453069    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:14.453055887 +0000 UTC m=+66.219017743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:12.453112 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:12.453082    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:14.453075843 +0000 UTC m=+66.219037700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : configmap "kube-rbac-proxy" not found
Jul 30 20:58:12.524039 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.524002    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:12.967136 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:12.967105    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s2nhn_2a4ce903-51af-4a3b-a1be-ae00c1bccb0b/dns-node-resolver/0.log"
Jul 30 20:58:13.967376 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:13.967353    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q5vql_7e57f05e-01a1-43ac-abb3-32f6daed266e/node-ca/0.log"
Jul 30 20:58:14.049460 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.049427    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:14.150214 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.150184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:14.260865 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.260839    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:14.370855 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.370825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:14.469796 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.469765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:58:14.469942 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.469867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:14.469942 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.469919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:58:14.470053 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.469969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:14.470053 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.469990    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 20:58:14.470164 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.470062    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:18.47004075 +0000 UTC m=+70.236002610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : configmap "kube-rbac-proxy" not found
Jul 30 20:58:14.470164 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.470074    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:14.470164 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.470122    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:18.470107937 +0000 UTC m=+70.236069796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:14.471927 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.471904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 30 20:58:14.472055 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.471992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 30 20:58:14.480420 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.480400    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 30 20:58:14.480547 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:14.480464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs podName:8a763470-0568-4720-a87c-636745df6053 nodeName:}" failed. No retries permitted until 2026-07-30 20:59:18.480447709 +0000 UTC m=+130.246409568 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs") pod "network-metrics-daemon-nlj6v" (UID: "8a763470-0568-4720-a87c-636745df6053") : secret "metrics-daemon-secret" not found
Jul 30 20:58:14.483180 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.483159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfa27313-a228-4ddc-abc6-78cb031f2476-original-pull-secret\") pod \"global-pull-secret-syncer-gts47\" (UID: \"dfa27313-a228-4ddc-abc6-78cb031f2476\") " pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:58:14.488099 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.488081    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gts47"
Jul 30 20:58:14.570590 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.570532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:58:14.579292 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.579262    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-flp9x_edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2/serve-healthcheck-canary/0.log"
Jul 30 20:58:14.580528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.580509    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 30 20:58:14.583926 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.583901    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 30 20:58:14.594175 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.594152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bszj4\" (UniqueName: \"kubernetes.io/projected/e0f6e363-dd36-41f1-8089-e003ab134549-kube-api-access-bszj4\") pod \"network-check-target-rxstk\" (UID: \"e0f6e363-dd36-41f1-8089-e003ab134549\") " pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:58:14.621486 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.621457    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gts47"]
Jul 30 20:58:14.624808 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:14.624784    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa27313_a228_4ddc_abc6_78cb031f2476.slice/crio-62f3371c64c17dd313fde655b0e0b9166be3ed87d07c360beeada95435db046d WatchSource:0}: Error finding container 62f3371c64c17dd313fde655b0e0b9166be3ed87d07c360beeada95435db046d: Status 404 returned error can't find the container with id 62f3371c64c17dd313fde655b0e0b9166be3ed87d07c360beeada95435db046d
Jul 30 20:58:14.797167 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.797085    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-66vkk\""
Jul 30 20:58:14.805620 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.805602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:58:14.933072 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:14.933039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rxstk"]
Jul 30 20:58:14.936146 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:14.936122    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f6e363_dd36_41f1_8089_e003ab134549.slice/crio-bac6cfb25e589e2768ad646735acdc0d3d6541633db11dcdf227e23ad0b76c14 WatchSource:0}: Error finding container bac6cfb25e589e2768ad646735acdc0d3d6541633db11dcdf227e23ad0b76c14: Status 404 returned error can't find the container with id bac6cfb25e589e2768ad646735acdc0d3d6541633db11dcdf227e23ad0b76c14
Jul 30 20:58:15.027525 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:15.027491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:15.126037 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:15.125956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxstk" event={"ID":"e0f6e363-dd36-41f1-8089-e003ab134549","Type":"ContainerStarted","Data":"bac6cfb25e589e2768ad646735acdc0d3d6541633db11dcdf227e23ad0b76c14"}
Jul 30 20:58:15.126944 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:15.126919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gts47" event={"ID":"dfa27313-a228-4ddc-abc6-78cb031f2476","Type":"ContainerStarted","Data":"62f3371c64c17dd313fde655b0e0b9166be3ed87d07c360beeada95435db046d"}
Jul 30 20:58:15.478813 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:15.478769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:58:15.478984 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:15.478891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:58:15.479103 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:15.479049    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 30 20:58:15.479149 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:15.479110    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79669ddfd9-qm598: secret "image-registry-tls" not found
Jul 30 20:58:15.479184 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:15.479171    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls podName:b3ba97b2-069e-4855-b664-70191df1498c nodeName:}" failed. No retries permitted until 2026-07-30 20:58:47.479152241 +0000 UTC m=+99.245114107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls") pod "image-registry-79669ddfd9-qm598" (UID: "b3ba97b2-069e-4855-b664-70191df1498c") : secret "image-registry-tls" not found
Jul 30 20:58:15.479603 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:15.479586    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 30 20:58:15.479714 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:15.479635    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls podName:9a5fc1e5-6b71-4156-9cd8-cda500ec1c22 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:47.479620121 +0000 UTC m=+99.245581999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls") pod "dns-default-zl2hb" (UID: "9a5fc1e5-6b71-4156-9cd8-cda500ec1c22") : secret "dns-default-metrics-tls" not found
Jul 30 20:58:17.977758 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:17.977726    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:18.505031 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:18.504993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:18.505220 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:18.505097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:18.505220 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:18.505149    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:18.505220 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:18.505190    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 30 20:58:18.505349 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:18.505221    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:26.505203704 +0000 UTC m=+78.271165581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:18.505349 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:18.505239    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:26.505230915 +0000 UTC m=+78.271192773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : configmap "kube-rbac-proxy" not found
Jul 30 20:58:18.845325 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:18.845236    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:19.093128 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:19.093083    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:19.193274 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:19.193187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:19.247732 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:19.247690    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:19.538130 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:19.538096    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:20.146611 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.146578    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gts47" event={"ID":"dfa27313-a228-4ddc-abc6-78cb031f2476","Type":"ContainerStarted","Data":"3f01ede032a1665bf4ae3d765bc2774f2ce82789da0ad467ecda4e66b7709a60"}
Jul 30 20:58:20.148073 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.148051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rxstk" event={"ID":"e0f6e363-dd36-41f1-8089-e003ab134549","Type":"ContainerStarted","Data":"003ff4ca929d89b8b3fad872a99012064bf70871c2b9127f61f8424928bd7f90"}
Jul 30 20:58:20.148199 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.148182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:58:20.159306 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.159267    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gts47" podStartSLOduration=66.227309748 podStartE2EDuration="1m11.159253955s" totalImagesPullingTime="4.931944207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:20.157961798 +0000 UTC m=+71.923923677" watchObservedRunningTime="2026-07-30 20:58:20.159253955 +0000 UTC m=+71.925215834"
Jul 30 20:58:20.171948 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.171907    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rxstk" podStartSLOduration=66.556099551 podStartE2EDuration="1m11.171893754s" totalImagesPullingTime="4.615794203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:20.171273179 +0000 UTC m=+71.937235059" watchObservedRunningTime="2026-07-30 20:58:20.171893754 +0000 UTC m=+71.937855661"
Jul 30 20:58:20.552181 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.552153    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:20.935082 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:20.934992    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:21.130044 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:21.130006    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:21.327165 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:21.327131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:21.427646 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:21.427616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:21.531099 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:21.531061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:22.476691 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:22.476658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:22.577473 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:22.577442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:22.732838 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:22.732760    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:22.946376 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:22.946345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:23.136102 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:23.136026    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:23.331977 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:23.331948    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:23.433669 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:23.433584    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:23.535426 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:23.535394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:23.981013 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:23.980984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:24.733331 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:24.733298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:24.932445 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:24.932414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:25.130210 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:25.130128    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:25.331586 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:25.331536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:25.432688 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:25.432615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:25.537707 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:25.537677    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:26.532046 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:26.532011    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:26.570889 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:26.570863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:26.571047 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:26.570919    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:26.571047 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:26.571022    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:26.571130 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:26.571085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls podName:dcf45e82-3aab-44db-a754-efe881059ce7 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:42.571071423 +0000 UTC m=+94.337033279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f9glr" (UID: "dcf45e82-3aab-44db-a754-efe881059ce7") : secret "insights-runtime-extractor-tls" not found
Jul 30 20:58:26.571365 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:26.571344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dcf45e82-3aab-44db-a754-efe881059ce7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:26.931796 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:26.931714    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:27.133512 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:27.133473    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:27.348220 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:27.348177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:27.534441 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:27.534403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:28.341766 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:28.341732    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:28.733110 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:28.733084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:28.933726 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:28.933694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:29.328199 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:29.328162    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:29.528124 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:29.528087    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:34.786512 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:34.786394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:34.911816 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:34.911786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:35.012803 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:35.012762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:36.844352 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:36.844314    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:36.898519 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:36.898487    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:39.839786 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:39.839747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:39.940852 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:39.940821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:40.003460 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:40.003427    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:40.109464 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:40.109378    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:42.033919 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.033882    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:42.125007 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.124970    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:42.303808 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.303729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:42.594901 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.594827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:42.597263 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.597242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dcf45e82-3aab-44db-a754-efe881059ce7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f9glr\" (UID: \"dcf45e82-3aab-44db-a754-efe881059ce7\") " pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:42.616272 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.616237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f9glr"
Jul 30 20:58:42.742892 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:42.742853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f9glr"]
Jul 30 20:58:42.746849 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:42.746815    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf45e82_3aab_44db_a754_efe881059ce7.slice/crio-2babc569b09c32b3f6f8d246ae3005244b5df5c7f8a7e99512eb710b414471da WatchSource:0}: Error finding container 2babc569b09c32b3f6f8d246ae3005244b5df5c7f8a7e99512eb710b414471da: Status 404 returned error can't find the container with id 2babc569b09c32b3f6f8d246ae3005244b5df5c7f8a7e99512eb710b414471da
Jul 30 20:58:43.219110 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:43.219077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9glr" event={"ID":"dcf45e82-3aab-44db-a754-efe881059ce7","Type":"ContainerStarted","Data":"7180eb8e294d3d689a0a5800dbfbfc3885eb349fc364e626bada3b4257245f12"}
Jul 30 20:58:43.219489 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:43.219135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9glr" event={"ID":"dcf45e82-3aab-44db-a754-efe881059ce7","Type":"ContainerStarted","Data":"2babc569b09c32b3f6f8d246ae3005244b5df5c7f8a7e99512eb710b414471da"}
Jul 30 20:58:44.224528 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:44.224477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9glr" event={"ID":"dcf45e82-3aab-44db-a754-efe881059ce7","Type":"ContainerStarted","Data":"d52413ee26fc2d24f89ba195d155aa7caf2c4932eb46b593c0f4d619b6e2c801"}
Jul 30 20:58:44.333051 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:44.333012    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:45.229584 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:45.229539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f9glr" event={"ID":"dcf45e82-3aab-44db-a754-efe881059ce7","Type":"ContainerStarted","Data":"cfd1cddbc2530b141fd93195939c983cb0458c9b75c78dfb058a8d831ba99bab"}
Jul 30 20:58:45.247779 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:45.247729    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f9glr" podStartSLOduration=33.153839967 podStartE2EDuration="35.247712451s" totalImagesPullingTime="2.093872484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 20:58:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:45.246925713 +0000 UTC m=+97.012887596" watchObservedRunningTime="2026-07-30 20:58:45.247712451 +0000 UTC m=+97.013674329"
Jul 30 20:58:46.172731 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.172704    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.276039 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.276008    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.404965 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.404927    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.457732 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.457647    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s4546"]
Jul 30 20:58:46.459704 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.459689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.461715 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.461688    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 30 20:58:46.461715 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.461702    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 30 20:58:46.461928 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.461919    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6shkw\""
Jul 30 20:58:46.462096 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.462062    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 30 20:58:46.462214 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.462104    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 30 20:58:46.462214 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.462180    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 30 20:58:46.462352 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.462335    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 30 20:58:46.507142 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.507111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.588961 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.588924    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.627271 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-sys\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627421 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627421 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627421 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627367    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-metrics-client-ca\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627421 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-textfile\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627662 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627491    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-root\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627662 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-wtmp\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627662 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwlr\" (UniqueName: \"kubernetes.io/projected/d12048cd-30bf-4440-800d-d3b07f30aec2-kube-api-access-gzwlr\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.627662 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.627607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.705814 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.705748    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.728623 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-textfile\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.728758 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-root\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.728758 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-wtmp\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.728758 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-root\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.728758 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwlr\" (UniqueName: \"kubernetes.io/projected/d12048cd-30bf-4440-800d-d3b07f30aec2-kube-api-access-gzwlr\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-sys\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728856    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-wtmp\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728853    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:46.728944    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d12048cd-30bf-4440-800d-d3b07f30aec2-sys\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-textfile\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.728964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-metrics-client-ca\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729004 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:58:46.728995    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls podName:d12048cd-30bf-4440-800d-d3b07f30aec2 nodeName:}" failed. No retries permitted until 2026-07-30 20:58:47.228978496 +0000 UTC m=+98.994940355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls") pod "node-exporter-s4546" (UID: "d12048cd-30bf-4440-800d-d3b07f30aec2") : secret "node-exporter-tls" not found
Jul 30 20:58:46.729380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.729295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.729416 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.729398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d12048cd-30bf-4440-800d-d3b07f30aec2-metrics-client-ca\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.731288 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.731271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.736616 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.736595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwlr\" (UniqueName: \"kubernetes.io/projected/d12048cd-30bf-4440-800d-d3b07f30aec2-kube-api-access-gzwlr\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:46.807492 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.807454    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:46.906754 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:46.906717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:47.101325 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.101244    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:47.234036 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.234009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:47.236845 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.236813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d12048cd-30bf-4440-800d-d3b07f30aec2-node-exporter-tls\") pod \"node-exporter-s4546\" (UID: \"d12048cd-30bf-4440-800d-d3b07f30aec2\") " pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:47.368883 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.368800    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s4546"
Jul 30 20:58:47.379118 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:47.379090    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12048cd_30bf_4440_800d_d3b07f30aec2.slice/crio-d3d23a7f2a925141135d49e71057ce80c02ec10010c8d83afcb13632f3392d0e WatchSource:0}: Error finding container d3d23a7f2a925141135d49e71057ce80c02ec10010c8d83afcb13632f3392d0e: Status 404 returned error can't find the container with id d3d23a7f2a925141135d49e71057ce80c02ec10010c8d83afcb13632f3392d0e
Jul 30 20:58:47.422160 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.422126    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:47.505994 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.505960    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:47.535692 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.535660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:58:47.535840 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.535719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:58:47.538093 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.538063    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9a5fc1e5-6b71-4156-9cd8-cda500ec1c22-metrics-tls\") pod \"dns-default-zl2hb\" (UID: \"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22\") " pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:58:47.538197 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.538093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"image-registry-79669ddfd9-qm598\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") " pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:58:47.557773 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.557749    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-77cz9\""
Jul 30 20:58:47.566104 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.566086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:58:47.622764 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.622693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:47.692725 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.692692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zl2hb"]
Jul 30 20:58:47.695817 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:47.695787    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5fc1e5_6b71_4156_9cd8_cda500ec1c22.slice/crio-681122725c7d82cf9c8bc0fbf46f9285c708eef8873740600405a3a983a2e07a WatchSource:0}: Error finding container 681122725c7d82cf9c8bc0fbf46f9285c708eef8873740600405a3a983a2e07a: Status 404 returned error can't find the container with id 681122725c7d82cf9c8bc0fbf46f9285c708eef8873740600405a3a983a2e07a
Jul 30 20:58:47.793879 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.793844    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-klzqk\""
Jul 30 20:58:47.802705 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.802683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:58:47.930624 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:47.930595    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:58:47.933485 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:58:47.933461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ba97b2_069e_4855_b664_70191df1498c.slice/crio-27a8e641675c189121692738eff790085d34af70dada99485578526dc3b05562 WatchSource:0}: Error finding container 27a8e641675c189121692738eff790085d34af70dada99485578526dc3b05562: Status 404 returned error can't find the container with id 27a8e641675c189121692738eff790085d34af70dada99485578526dc3b05562
Jul 30 20:58:48.243281 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.243236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" event={"ID":"b3ba97b2-069e-4855-b664-70191df1498c","Type":"ContainerStarted","Data":"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"}
Jul 30 20:58:48.243281 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.243286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" event={"ID":"b3ba97b2-069e-4855-b664-70191df1498c","Type":"ContainerStarted","Data":"27a8e641675c189121692738eff790085d34af70dada99485578526dc3b05562"}
Jul 30 20:58:48.243525 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.243352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:58:48.244589 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.244539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zl2hb" event={"ID":"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22","Type":"ContainerStarted","Data":"681122725c7d82cf9c8bc0fbf46f9285c708eef8873740600405a3a983a2e07a"}
Jul 30 20:58:48.246269 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.246211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4546" event={"ID":"d12048cd-30bf-4440-800d-d3b07f30aec2","Type":"ContainerStarted","Data":"d3d23a7f2a925141135d49e71057ce80c02ec10010c8d83afcb13632f3392d0e"}
Jul 30 20:58:48.264622 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:48.264542    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" podStartSLOduration=99.264520812 podStartE2EDuration="1m39.264520812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:48.26341322 +0000 UTC m=+100.029375100" watchObservedRunningTime="2026-07-30 20:58:48.264520812 +0000 UTC m=+100.030482695"
Jul 30 20:58:49.251430 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:49.251395    2583 generic.go:356] "Generic (PLEG): container finished" podID="d12048cd-30bf-4440-800d-d3b07f30aec2" containerID="db6e81dbc2e83b1dfc2eb1b377c7398f25d17b9477f558220980df032755d94d" exitCode=0
Jul 30 20:58:49.251927 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:49.251476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4546" event={"ID":"d12048cd-30bf-4440-800d-d3b07f30aec2","Type":"ContainerDied","Data":"db6e81dbc2e83b1dfc2eb1b377c7398f25d17b9477f558220980df032755d94d"}
Jul 30 20:58:49.377748 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:49.377693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:49.478582 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:49.478544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:49.546651 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:49.546552    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:50.082990 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.082966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:50.256938 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.256897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zl2hb" event={"ID":"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22","Type":"ContainerStarted","Data":"dc6b89354d02c41207cf48bf86322ed967bb31c36bda2479302fa777f2b708ba"}
Jul 30 20:58:50.257362 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.256996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zl2hb" event={"ID":"9a5fc1e5-6b71-4156-9cd8-cda500ec1c22","Type":"ContainerStarted","Data":"6aa2c68f0108767c6b8b3876b811c9a17f7362bc1a817f6fc5677ce39e788ee8"}
Jul 30 20:58:50.257362 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.257028    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:58:50.259054 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.259027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4546" event={"ID":"d12048cd-30bf-4440-800d-d3b07f30aec2","Type":"ContainerStarted","Data":"41b2865c44dae58d490d9ce2962d1ce211c11b95af2e028fe2c4ce914bf58d13"}
Jul 30 20:58:50.259182 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.259075    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s4546" event={"ID":"d12048cd-30bf-4440-800d-d3b07f30aec2","Type":"ContainerStarted","Data":"6c812064ce28c918f82a9f9776931a8acb48897ca2247bc4eb66436a99ab1bd9"}
Jul 30 20:58:50.273420 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.273373    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zl2hb" podStartSLOduration=65.128580965 podStartE2EDuration="1m7.273358102s" totalImagesPullingTime="2.144777137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:50.272518074 +0000 UTC m=+102.038479952" watchObservedRunningTime="2026-07-30 20:58:50.273358102 +0000 UTC m=+102.039319981"
Jul 30 20:58:50.289702 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:50.289653    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s4546" podStartSLOduration=3.13897967 podStartE2EDuration="4.289638251s" totalImagesPullingTime="1.075565268s" totalInitContainerRuntime="75.093313ms" isStatefulPod=true podCreationTimestamp="2026-07-30 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:58:50.288722796 +0000 UTC m=+102.054684700" watchObservedRunningTime="2026-07-30 20:58:50.289638251 +0000 UTC m=+102.055600129"
Jul 30 20:58:51.154116 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:51.154085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rxstk"
Jul 30 20:58:51.190961 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:51.190931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:51.482120 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:51.482088    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:52.479767 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:52.479736    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:52.678279 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:52.678252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:56.788619 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:56.788590    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:58:56.872456 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:56.872422    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:56.972716 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:56.972686    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:58:57.021867 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:58:57.021836    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:00.264734 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:00.264703    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zl2hb"
Jul 30 20:59:05.734127 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:05.734093    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:05.798953 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:05.798915    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:06.003061 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.002985    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:06.371681 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.371582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:06.769660 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.769627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:06.794240 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.794212    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:59:06.867876 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.867847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:06.920257 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:06.920218    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:07.405397 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:07.405358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:07.608865 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:07.608832    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:07.812676 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:07.812648    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:07.913458 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:07.913427    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:08.425644 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:08.425614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:08.480224 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:08.480196    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:09.130974 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:09.130939    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:09.191372 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:09.191345    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:09.391036 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:09.390950    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:09.448030 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:09.447983    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:14.187857 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:14.187821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:14.242483 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:14.242452    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:15.883179 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:15.883145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:15.935556 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:15.935525    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:17.075189 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:17.075139    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:17.135047 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:17.135009    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:18.390685 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.390649    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:18.455073 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.455035    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:18.568768 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.568734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:59:18.571222 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.571198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a763470-0568-4720-a87c-636745df6053-metrics-certs\") pod \"network-metrics-daemon-nlj6v\" (UID: \"8a763470-0568-4720-a87c-636745df6053\") " pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:59:18.589456 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.589426    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:18.704180 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.704095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xh5pq\""
Jul 30 20:59:18.712601 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.712584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nlj6v"
Jul 30 20:59:18.843389 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:18.843356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nlj6v"]
Jul 30 20:59:18.846424 ip-10-0-143-139 kubenswrapper[2583]: W0730 20:59:18.846390    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a763470_0568_4720_a87c_636745df6053.slice/crio-b449cc359c8cc654f66772c5e8fdeeac522d5767684b1bf3cf1049085f61e750 WatchSource:0}: Error finding container b449cc359c8cc654f66772c5e8fdeeac522d5767684b1bf3cf1049085f61e750: Status 404 returned error can't find the container with id b449cc359c8cc654f66772c5e8fdeeac522d5767684b1bf3cf1049085f61e750
Jul 30 20:59:19.356419 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:19.356375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlj6v" event={"ID":"8a763470-0568-4720-a87c-636745df6053","Type":"ContainerStarted","Data":"b449cc359c8cc654f66772c5e8fdeeac522d5767684b1bf3cf1049085f61e750"}
Jul 30 20:59:20.362310 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:20.362271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlj6v" event={"ID":"8a763470-0568-4720-a87c-636745df6053","Type":"ContainerStarted","Data":"dfbe77ecb980a82752ec51fd82f5b66546aab3870e19417bf304e11c2f38d83f"}
Jul 30 20:59:21.367688 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.367652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nlj6v" event={"ID":"8a763470-0568-4720-a87c-636745df6053","Type":"ContainerStarted","Data":"3c5a02b504ee52e04561be5761bfb69495616e73099b3e6a630640fe8e926cde"}
Jul 30 20:59:21.383446 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.383379    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nlj6v" podStartSLOduration=131.009993991 podStartE2EDuration="2m12.383357008s" totalImagesPullingTime="1.373363017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 20:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 20:59:21.381611562 +0000 UTC m=+133.147573441" watchObservedRunningTime="2026-07-30 20:59:21.383357008 +0000 UTC m=+133.149318891"
Jul 30 20:59:21.445168 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.445133    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:21.545022 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.544971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:21.596508 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.596459    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:21.807479 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:21.807438    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" podUID="b3ba97b2-069e-4855-b664-70191df1498c" containerName="registry" containerID="cri-o://06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203" gracePeriod=30
Jul 30 20:59:22.037540 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.037512    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:59:22.095405 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095405 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095370    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095405    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095442    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095470    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095502    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlxpj\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095627 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095533    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095911 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095654    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") pod \"b3ba97b2-069e-4855-b664-70191df1498c\" (UID: \"b3ba97b2-069e-4855-b664-70191df1498c\") "
Jul 30 20:59:22.095911 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095777    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 20:59:22.095911 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095881    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-registry-certificates\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.096040 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.095957    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 20:59:22.098283 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.098250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 20:59:22.098457 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.098420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj" (OuterVolumeSpecName: "kube-api-access-vlxpj") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "kube-api-access-vlxpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 20:59:22.098590 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.098457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 20:59:22.098590 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.098516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 20:59:22.098590 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.098549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 20:59:22.103885 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.103862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3ba97b2-069e-4855-b664-70191df1498c" (UID: "b3ba97b2-069e-4855-b664-70191df1498c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 20:59:22.196173 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196143    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxpj\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-kube-api-access-vlxpj\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196173 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196167    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3ba97b2-069e-4855-b664-70191df1498c-ca-trust-extracted\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196173 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196177    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-registry-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196188    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-image-registry-private-configuration\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196199    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3ba97b2-069e-4855-b664-70191df1498c-trusted-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196207    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3ba97b2-069e-4855-b664-70191df1498c-installation-pull-secrets\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.196380 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.196214    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ba97b2-069e-4855-b664-70191df1498c-bound-sa-token\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 20:59:22.372085 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.371999    2583 generic.go:356] "Generic (PLEG): container finished" podID="b3ba97b2-069e-4855-b664-70191df1498c" containerID="06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203" exitCode=0
Jul 30 20:59:22.372085 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.372047    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79669ddfd9-qm598"
Jul 30 20:59:22.372516 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.372093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" event={"ID":"b3ba97b2-069e-4855-b664-70191df1498c","Type":"ContainerDied","Data":"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"}
Jul 30 20:59:22.372516 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.372175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79669ddfd9-qm598" event={"ID":"b3ba97b2-069e-4855-b664-70191df1498c","Type":"ContainerDied","Data":"27a8e641675c189121692738eff790085d34af70dada99485578526dc3b05562"}
Jul 30 20:59:22.372516 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.372208    2583 scope.go:117] "RemoveContainer" containerID="06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"
Jul 30 20:59:22.381149 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.381127    2583 scope.go:117] "RemoveContainer" containerID="06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"
Jul 30 20:59:22.381410 ip-10-0-143-139 kubenswrapper[2583]: E0730 20:59:22.381390    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203\": container with ID starting with 06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203 not found: ID does not exist" containerID="06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"
Jul 30 20:59:22.381490 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.381418    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203"} err="failed to get container status \"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203\": rpc error: code = NotFound desc = could not find container \"06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203\": container with ID starting with 06b83ca2a3e81be277e355f57eaa496b3f84d444483f6013663cf78bd2076203 not found: ID does not exist"
Jul 30 20:59:22.390589 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.390550    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:59:22.394283 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.394264    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79669ddfd9-qm598"]
Jul 30 20:59:22.775581 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:22.775536    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ba97b2-069e-4855-b664-70191df1498c" path="/var/lib/kubelet/pods/b3ba97b2-069e-4855-b664-70191df1498c/volumes"
Jul 30 20:59:23.910901 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:23.910848    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" podUID="8e791680-1811-4ace-a28c-49f1ebd6b997" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 20:59:24.637635 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:24.637608    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:24.691882 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:24.691850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:25.267132 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:25.267099    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:25.321289 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:25.321250    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:26.652466 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:26.652431    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-flp9x_edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2/serve-healthcheck-canary/0.log"
Jul 30 20:59:28.626347 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:28.626313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:30.228722 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:30.228691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:30.278007 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:30.277975    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:33.909994 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:33.909955    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" podUID="8e791680-1811-4ace-a28c-49f1ebd6b997" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 20:59:34.030214 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:34.030177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:34.087225 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:34.087191    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:36.707246 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:36.707213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:36.838598 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:36.838548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:36.897498 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:36.897454    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:37.455090 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:37.455057    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:40.004618 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:40.004560    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:40.152336 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:40.152305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:40.254357 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:40.254321    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:43.909993 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:43.909943    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" podUID="8e791680-1811-4ace-a28c-49f1ebd6b997" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 30 20:59:43.910367 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:43.910036    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5"
Jul 30 20:59:43.914376 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:43.914340    2583 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"aa607eef0d2181b23f3864c0650c12a0c36e69ec0913d5842f6fc4a8e499edde"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 30 20:59:43.914503 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:43.914393    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" podUID="8e791680-1811-4ace-a28c-49f1ebd6b997" containerName="service-proxy" containerID="cri-o://aa607eef0d2181b23f3864c0650c12a0c36e69ec0913d5842f6fc4a8e499edde" gracePeriod=30
Jul 30 20:59:44.450682 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:44.450649    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e791680-1811-4ace-a28c-49f1ebd6b997" containerID="aa607eef0d2181b23f3864c0650c12a0c36e69ec0913d5842f6fc4a8e499edde" exitCode=2
Jul 30 20:59:44.450846 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:44.450720    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerDied","Data":"aa607eef0d2181b23f3864c0650c12a0c36e69ec0913d5842f6fc4a8e499edde"}
Jul 30 20:59:44.450846 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:44.450767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6549dc97ff-rcrp5" event={"ID":"8e791680-1811-4ace-a28c-49f1ebd6b997","Type":"ContainerStarted","Data":"bcdf5a2cb123798241785f35da13ecdf9f32a0a04b9e461b9c1144fc09b4cbdd"}
Jul 30 20:59:51.849707 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:51.849674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 20:59:51.914988 ip-10-0-143-139 kubenswrapper[2583]: I0730 20:59:51.914955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:02.546816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:02.546744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:02.748447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:02.748412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:03.905815 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:03.905781    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:03.963619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:03.963585    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:04.263917 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:04.263885    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:14.569395 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:14.569358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:14.627075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:14.627042    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:14.872442 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:14.872347    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:14.973011 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:14.972975    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:15.047616 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:15.047582    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:21.840898 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:21.840860    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:21.941778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:21.941744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:38.524581 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:38.524539    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:56.129969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:56.129926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:56.223946 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:56.223901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:00:56.746300 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:00:56.746261    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:02.689169 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:02.689139    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:02.742178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:02.742142    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:27.962991 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:27.962955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:28.739959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:28.739927    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:31.562853 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:31.562767    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:31.667791 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:31.667755    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:36.357905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:36.357868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:36.412532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:36.412491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:44.350733 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:44.350692    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:44.453981 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:44.453940    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:44.579911 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:44.579877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:44.695799 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:44.695721    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:46.514417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:46.514376    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:46.916191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:46.916107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:53.378046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:53.378004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:53.432960 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:53.432930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:54.517249 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:54.517216    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:55.270712 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.270679    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cnhs9"]
Jul 30 21:01:55.270997 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.270980    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ba97b2-069e-4855-b664-70191df1498c" containerName="registry"
Jul 30 21:01:55.271072 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.271000    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ba97b2-069e-4855-b664-70191df1498c" containerName="registry"
Jul 30 21:01:55.271072 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.271068    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ba97b2-069e-4855-b664-70191df1498c" containerName="registry"
Jul 30 21:01:55.273789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.273769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.275736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.275719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 30 21:01:55.275812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.275784    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-v7rvf\""
Jul 30 21:01:55.275866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.275852    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 30 21:01:55.283375 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.283354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cnhs9"]
Jul 30 21:01:55.349682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.349646    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg2lr\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-kube-api-access-wg2lr\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.349830 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.349696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.450976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.450930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.451156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.450994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg2lr\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-kube-api-access-wg2lr\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.458798 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.458770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-bound-sa-token\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.458905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.458875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg2lr\" (UniqueName: \"kubernetes.io/projected/65bcdcf9-75e2-41bd-938b-daeae7af2f5a-kube-api-access-wg2lr\") pod \"cert-manager-55b4d954d5-cnhs9\" (UID: \"65bcdcf9-75e2-41bd-938b-daeae7af2f5a\") " pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.583188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.583084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-cnhs9"
Jul 30 21:01:55.710928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.710895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-cnhs9"]
Jul 30 21:01:55.714298 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:01:55.714273    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65bcdcf9_75e2_41bd_938b_daeae7af2f5a.slice/crio-094ce4c23243354d707ad661cacc7821856f180b783a369d40dd60ac2895d9ec WatchSource:0}: Error finding container 094ce4c23243354d707ad661cacc7821856f180b783a369d40dd60ac2895d9ec: Status 404 returned error can't find the container with id 094ce4c23243354d707ad661cacc7821856f180b783a369d40dd60ac2895d9ec
Jul 30 21:01:55.855898 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:55.855809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cnhs9" event={"ID":"65bcdcf9-75e2-41bd-938b-daeae7af2f5a","Type":"ContainerStarted","Data":"094ce4c23243354d707ad661cacc7821856f180b783a369d40dd60ac2895d9ec"}
Jul 30 21:01:57.875955 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:57.875906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:57.975474 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:57.975387    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:01:59.874027 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:59.873987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-cnhs9" event={"ID":"65bcdcf9-75e2-41bd-938b-daeae7af2f5a","Type":"ContainerStarted","Data":"53ec7639a97e8c5a2e6c84fe03ecfa225383502b1c4a790d69b71185c12ed549"}
Jul 30 21:01:59.892492 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:01:59.892444    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-cnhs9" podStartSLOduration=1.501910871 podStartE2EDuration="4.892430681s" totalImagesPullingTime="3.39051981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:01:59.891060068 +0000 UTC m=+291.657021947" watchObservedRunningTime="2026-07-30 21:01:59.892430681 +0000 UTC m=+291.658392560"
Jul 30 21:02:02.886398 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:02.886338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:03.355208 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:03.355177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:06.837446 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:06.837413    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:06.950040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:06.950004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:08.654481 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:08.654448    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:02:08.655560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:08.655537    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:02:08.660798 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:08.660777    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 30 21:02:09.440602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:09.440548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:19.704545 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:19.704510    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:19.816255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:19.816223    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:20.647151 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:20.647119    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:20.752175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:20.751848    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:21.246466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.246421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:21.442796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.442756    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:21.543397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.543311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:21.581402 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.581376    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"]
Jul 30 21:02:21.585927 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.585911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.588493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.588471    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 30 21:02:21.588764 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.588744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 30 21:02:21.588850 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.588752    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dq824\""
Jul 30 21:02:21.588992 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.588975    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 30 21:02:21.589706 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.589692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 30 21:02:21.701500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.701464    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"]
Jul 30 21:02:21.726181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.726139    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.726181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.726180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.726443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.726204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dfqm\" (UniqueName: \"kubernetes.io/projected/1543b89f-afdb-430f-9be1-ac2b362e2ee7-kube-api-access-6dfqm\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.726443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.726310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.726443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.726362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.815374 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.815296    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:21.827598 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.827687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.827687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.827687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dfqm\" (UniqueName: \"kubernetes.io/projected/1543b89f-afdb-430f-9be1-ac2b362e2ee7-kube-api-access-6dfqm\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.827833 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.828017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.827996    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-manifests\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.830178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.830140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1543b89f-afdb-430f-9be1-ac2b362e2ee7-tmp\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.830279 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.830200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.830321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.830304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1543b89f-afdb-430f-9be1-ac2b362e2ee7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.836874 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.836853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dfqm\" (UniqueName: \"kubernetes.io/projected/1543b89f-afdb-430f-9be1-ac2b362e2ee7-kube-api-access-6dfqm\") pod \"opendatahub-operator-controller-manager-5ddcd94768-fdhfj\" (UID: \"1543b89f-afdb-430f-9be1-ac2b362e2ee7\") " pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.896070 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.896033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:21.916588 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:21.916537    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:22.016552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.016523    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:22.035102 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.034971    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"]
Jul 30 21:02:22.037768 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:02:22.037746    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1543b89f_afdb_430f_9be1_ac2b362e2ee7.slice/crio-6a283845042f759911624cdd2c12a70a844d68794031c8f0c87f0240b5417c7f WatchSource:0}: Error finding container 6a283845042f759911624cdd2c12a70a844d68794031c8f0c87f0240b5417c7f: Status 404 returned error can't find the container with id 6a283845042f759911624cdd2c12a70a844d68794031c8f0c87f0240b5417c7f
Jul 30 21:02:22.039484 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.039467    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:02:22.115696 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.115607    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:22.216853 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.216820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:22.307313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.307267    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:22.949754 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:22.949703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj" event={"ID":"1543b89f-afdb-430f-9be1-ac2b362e2ee7","Type":"ContainerStarted","Data":"6a283845042f759911624cdd2c12a70a844d68794031c8f0c87f0240b5417c7f"}
Jul 30 21:02:25.966407 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:25.966371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj" event={"ID":"1543b89f-afdb-430f-9be1-ac2b362e2ee7","Type":"ContainerStarted","Data":"528ee2b83ec91e0beed8df9448de80d749f091870af0594859427d90804adc76"}
Jul 30 21:02:26.971696 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:26.971660    2583 generic.go:356] "Generic (PLEG): container finished" podID="1543b89f-afdb-430f-9be1-ac2b362e2ee7" containerID="528ee2b83ec91e0beed8df9448de80d749f091870af0594859427d90804adc76" exitCode=0
Jul 30 21:02:26.972102 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:26.971752    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj" event={"ID":"1543b89f-afdb-430f-9be1-ac2b362e2ee7","Type":"ContainerDied","Data":"528ee2b83ec91e0beed8df9448de80d749f091870af0594859427d90804adc76"}
Jul 30 21:02:27.976662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:27.976610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj" event={"ID":"1543b89f-afdb-430f-9be1-ac2b362e2ee7","Type":"ContainerStarted","Data":"7e46d5f64a5f47f7c13ac6e248760417f1542a054607329be66f1e8883286b7e"}
Jul 30 21:02:27.977136 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:27.976726    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:27.997526 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:27.997466    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj" podStartSLOduration=2.552941321 podStartE2EDuration="6.99744893s" totalImagesPullingTime="3.432167838s" totalInitContainerRuntime="1.012339771s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:27.997150276 +0000 UTC m=+319.763112157" watchObservedRunningTime="2026-07-30 21:02:27.99744893 +0000 UTC m=+319.763411184"
Jul 30 21:02:38.982249 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:38.982221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ddcd94768-fdhfj"
Jul 30 21:02:42.189537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:42.189503    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:44.218328 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:44.218297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.057052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.057019    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"]
Jul 30 21:02:53.057602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.057135    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.064322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.064301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.066761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.066737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 30 21:02:53.066761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.066752    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 30 21:02:53.066905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.066741    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 30 21:02:53.067040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.067025    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 30 21:02:53.075545 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.075521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"]
Jul 30 21:02:53.150633 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.150595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbvh\" (UniqueName: \"kubernetes.io/projected/a79443e1-d3bc-48dc-89b1-942ea2625008-kube-api-access-dmbvh\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.150633 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.150634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.150869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.150703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79443e1-d3bc-48dc-89b1-942ea2625008-tmp\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.251756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.251718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbvh\" (UniqueName: \"kubernetes.io/projected/a79443e1-d3bc-48dc-89b1-942ea2625008-kube-api-access-dmbvh\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.251756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.251760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.251965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.251784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79443e1-d3bc-48dc-89b1-942ea2625008-tmp\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.251965 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:02:53.251880    2583 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 30 21:02:53.251965 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:02:53.251960    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs podName:a79443e1-d3bc-48dc-89b1-942ea2625008 nodeName:}" failed. No retries permitted until 2026-07-30 21:02:53.751942562 +0000 UTC m=+345.517904420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs") pod "kube-auth-proxy-65ffdd497f-9dksv" (UID: "a79443e1-d3bc-48dc-89b1-942ea2625008") : secret "kube-auth-proxy-tls" not found
Jul 30 21:02:53.254247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.254226    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a79443e1-d3bc-48dc-89b1-942ea2625008-tmp\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.265278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.265258    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtlkp"]
Jul 30 21:02:53.268119 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.268086    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.271145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.271108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbvh\" (UniqueName: \"kubernetes.io/projected/a79443e1-d3bc-48dc-89b1-942ea2625008-kube-api-access-dmbvh\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.273079 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.273058    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 30 21:02:53.273248 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.273104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-zgdh4\""
Jul 30 21:02:53.283536 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.283510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtlkp"]
Jul 30 21:02:53.303400 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.303373    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.352793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.352715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3349474-912f-4c9b-bd42-4cda739d3231-cert\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.352793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.352768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx8b\" (UniqueName: \"kubernetes.io/projected/f3349474-912f-4c9b-bd42-4cda739d3231-kube-api-access-hqx8b\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.411785 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.411753    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.454201 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.454166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3349474-912f-4c9b-bd42-4cda739d3231-cert\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.454381 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.454219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx8b\" (UniqueName: \"kubernetes.io/projected/f3349474-912f-4c9b-bd42-4cda739d3231-kube-api-access-hqx8b\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.457078 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.457054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f3349474-912f-4c9b-bd42-4cda739d3231-cert\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.462466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.462441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx8b\" (UniqueName: \"kubernetes.io/projected/f3349474-912f-4c9b-bd42-4cda739d3231-kube-api-access-hqx8b\") pod \"odh-model-controller-7998c67dc4-dtlkp\" (UID: \"f3349474-912f-4c9b-bd42-4cda739d3231\") " pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.522635 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.522601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.586261 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.586216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:53.712186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.712151    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.722699 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.722665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-dtlkp"]
Jul 30 21:02:53.728393 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:02:53.728213    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3349474_912f_4c9b_bd42_4cda739d3231.slice/crio-fc170fd86f1f43e1c7d9497365b8d9d1aabe6ed3d12507d816fb23aa817c27c3 WatchSource:0}: Error finding container fc170fd86f1f43e1c7d9497365b8d9d1aabe6ed3d12507d816fb23aa817c27c3: Status 404 returned error can't find the container with id fc170fd86f1f43e1c7d9497365b8d9d1aabe6ed3d12507d816fb23aa817c27c3
Jul 30 21:02:53.756439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.756408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.758893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.758868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79443e1-d3bc-48dc-89b1-942ea2625008-tls-certs\") pod \"kube-auth-proxy-65ffdd497f-9dksv\" (UID: \"a79443e1-d3bc-48dc-89b1-942ea2625008\") " pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.782853 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.782827    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s4zw5"]
Jul 30 21:02:53.787096 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.787078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:53.789506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.789481    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 30 21:02:53.813463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.813434    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s4zw5"]
Jul 30 21:02:53.841540 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.841489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:53.958891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.958803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-tls-certs\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:53.959032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.958911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvhq\" (UniqueName: \"kubernetes.io/projected/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-kube-api-access-dgvhq\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:53.973770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.973749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"
Jul 30 21:02:53.987762 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:53.987717    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:54.060078 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.060049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvhq\" (UniqueName: \"kubernetes.io/projected/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-kube-api-access-dgvhq\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:54.060584 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.060087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-tls-certs\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:54.062739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.062712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-tls-certs\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:54.065207 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.065180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp" event={"ID":"f3349474-912f-4c9b-bd42-4cda739d3231","Type":"ContainerStarted","Data":"fc170fd86f1f43e1c7d9497365b8d9d1aabe6ed3d12507d816fb23aa817c27c3"}
Jul 30 21:02:54.068397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.068375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvhq\" (UniqueName: \"kubernetes.io/projected/dc3dfbc2-af6b-4bb6-ad41-3542189d2859-kube-api-access-dgvhq\") pod \"model-serving-api-54d89bd5fc-s4zw5\" (UID: \"dc3dfbc2-af6b-4bb6-ad41-3542189d2859\") " pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:54.097547 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.097510    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:54.108219 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.108189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv"]
Jul 30 21:02:54.111284 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:02:54.111254    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79443e1_d3bc_48dc_89b1_942ea2625008.slice/crio-f5ce1070179846b65dff44205e93e3023ee11cefe33a1cfd870e35b0146f0653 WatchSource:0}: Error finding container f5ce1070179846b65dff44205e93e3023ee11cefe33a1cfd870e35b0146f0653: Status 404 returned error can't find the container with id f5ce1070179846b65dff44205e93e3023ee11cefe33a1cfd870e35b0146f0653
Jul 30 21:02:54.238463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.238414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-s4zw5"]
Jul 30 21:02:54.242247 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:02:54.242219    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3dfbc2_af6b_4bb6_ad41_3542189d2859.slice/crio-17dc13d0b99100de10dd42a4503b56d108e567f0c0607b15db82a1cc28ff3165 WatchSource:0}: Error finding container 17dc13d0b99100de10dd42a4503b56d108e567f0c0607b15db82a1cc28ff3165: Status 404 returned error can't find the container with id 17dc13d0b99100de10dd42a4503b56d108e567f0c0607b15db82a1cc28ff3165
Jul 30 21:02:54.418029 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.417991    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:02:54.421199 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.421164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.423640 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.423612    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:54.423937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.423921    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 30 21:02:54.424189 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.424174    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 30 21:02:54.424760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.424395    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jgslc\""
Jul 30 21:02:54.424760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.424504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 30 21:02:54.424760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.424706    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 30 21:02:54.434978 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.434917    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:02:54.492501 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.492385    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:54.564987 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.564864    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.564987 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.564912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.564987 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.564946    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.565285 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.565055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.565285 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.565091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.565285 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.565140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.565285 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.565178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjdw\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.665933 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.665899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.665950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.665986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.666010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.666054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.666090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjdw\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666445 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.666156    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.666671 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.666642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.670675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.669367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.670675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.669598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.670675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.669940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.670675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.669986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.674450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.674407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.675450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.675424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjdw\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw\") pod \"istiod-openshift-gateway-7b7995d5cd-q9rhp\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.734422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.734384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:02:54.960994 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:54.960742    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:02:54.963482 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:02:54.963449    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5b3984_09e7_4f7d_b0a2_10c3d64003f0.slice/crio-c872a6a860e2011d7d53d2260ec7c3fff1d06f46fe1ccd57b2f80470dc9a3281 WatchSource:0}: Error finding container c872a6a860e2011d7d53d2260ec7c3fff1d06f46fe1ccd57b2f80470dc9a3281: Status 404 returned error can't find the container with id c872a6a860e2011d7d53d2260ec7c3fff1d06f46fe1ccd57b2f80470dc9a3281
Jul 30 21:02:55.072382 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:55.072294    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" event={"ID":"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0","Type":"ContainerStarted","Data":"c872a6a860e2011d7d53d2260ec7c3fff1d06f46fe1ccd57b2f80470dc9a3281"}
Jul 30 21:02:55.074229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:55.074202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv" event={"ID":"a79443e1-d3bc-48dc-89b1-942ea2625008","Type":"ContainerStarted","Data":"f5ce1070179846b65dff44205e93e3023ee11cefe33a1cfd870e35b0146f0653"}
Jul 30 21:02:55.075493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:55.075432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5" event={"ID":"dc3dfbc2-af6b-4bb6-ad41-3542189d2859","Type":"ContainerStarted","Data":"17dc13d0b99100de10dd42a4503b56d108e567f0c0607b15db82a1cc28ff3165"}
Jul 30 21:02:55.410028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:55.409902    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:02:58.089656 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.089545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv" event={"ID":"a79443e1-d3bc-48dc-89b1-942ea2625008","Type":"ContainerStarted","Data":"4f0043bb93783df1489d1efbec7972d23a2fb78f04200e65c5c2c49ff0a028d2"}
Jul 30 21:02:58.096181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.091827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5" event={"ID":"dc3dfbc2-af6b-4bb6-ad41-3542189d2859","Type":"ContainerStarted","Data":"7c010a842025828f9df34671e854e4499799761c692811b0cf0e876fdd9b1e5a"}
Jul 30 21:02:58.096181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.091902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:02:58.096551 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.096241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp" event={"ID":"f3349474-912f-4c9b-bd42-4cda739d3231","Type":"ContainerStarted","Data":"19d9a02ed21ee404f2a4ef2ca7c8e5402ad87debb69c85e3dbaba89080d588fd"}
Jul 30 21:02:58.096836 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.096777    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:02:58.135959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.133203    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp" podStartSLOduration=0.897658946 podStartE2EDuration="5.133178565s" totalImagesPullingTime="4.235519619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:58.127665917 +0000 UTC m=+349.893627804" watchObservedRunningTime="2026-07-30 21:02:58.133178565 +0000 UTC m=+349.899140444"
Jul 30 21:02:58.136428 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.136332    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65ffdd497f-9dksv" podStartSLOduration=1.270749663 podStartE2EDuration="5.136310881s" totalImagesPullingTime="3.865561218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:58.10847571 +0000 UTC m=+349.874437591" watchObservedRunningTime="2026-07-30 21:02:58.136310881 +0000 UTC m=+349.902272761"
Jul 30 21:02:58.150094 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:58.150032    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5" podStartSLOduration=1.427191555 podStartE2EDuration="5.150009769s" totalImagesPullingTime="3.722818214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:02:58.148070876 +0000 UTC m=+349.914032757" watchObservedRunningTime="2026-07-30 21:02:58.150009769 +0000 UTC m=+349.915971649"
Jul 30 21:02:59.432309 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:59.432270    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 30 21:02:59.432720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:02:59.432337    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 30 21:03:00.107767 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.107674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" event={"ID":"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0","Type":"ContainerStarted","Data":"444f0479fd41dfcd8bb81749d38d8f0608b9182cb6b84b50297acc6b95841249"}
Jul 30 21:03:00.108090 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.108067    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:03:00.109655 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.109626    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-q9rhp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 21:03:00.109770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.109683    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 21:03:00.127443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.127392    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" podStartSLOduration=1.661165104 podStartE2EDuration="6.127376556s" totalImagesPullingTime="4.466211452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:02:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:03:00.126457147 +0000 UTC m=+351.892419027" watchObservedRunningTime="2026-07-30 21:03:00.127376556 +0000 UTC m=+351.893338434"
Jul 30 21:03:00.537550 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:00.537515    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:01.113359 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:01.113325    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-q9rhp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 30 21:03:01.113659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:01.113619    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 30 21:03:01.184702 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:01.184664    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:01.377659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:01.377548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:01.476260 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:01.476220    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:04.113426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.113395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:03:04.681133 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.681090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:04.736888 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.736852    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"]
Jul 30 21:03:04.739632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.739610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.741800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.741780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 30 21:03:04.741911 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.741782    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-snv47\""
Jul 30 21:03:04.751056 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.751034    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"]
Jul 30 21:03:04.753794 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.753773    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.755452 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.755434    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"]
Jul 30 21:03:04.755707 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.755686    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 30 21:03:04.755800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.755711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hp6c2\""
Jul 30 21:03:04.767264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.767235    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"]
Jul 30 21:03:04.858105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.858070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4g4\" (UniqueName: \"kubernetes.io/projected/5090fc76-a383-4e9e-8a71-719d87ec90bd-kube-api-access-ws4g4\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.858277 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.858122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83e40ff0-3ce8-4f14-a555-364c465aa18a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.858277 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.858148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxx79\" (UniqueName: \"kubernetes.io/projected/83e40ff0-3ce8-4f14-a555-364c465aa18a-kube-api-access-zxx79\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.858277 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.858184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.959382 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.959293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83e40ff0-3ce8-4f14-a555-364c465aa18a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.959382 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.959343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxx79\" (UniqueName: \"kubernetes.io/projected/83e40ff0-3ce8-4f14-a555-364c465aa18a-kube-api-access-zxx79\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.959382 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.959388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.959639 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.959426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4g4\" (UniqueName: \"kubernetes.io/projected/5090fc76-a383-4e9e-8a71-719d87ec90bd-kube-api-access-ws4g4\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.959639 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:03:04.959535    2583 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 30 21:03:04.959639 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:03:04.959636    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert podName:5090fc76-a383-4e9e-8a71-719d87ec90bd nodeName:}" failed. No retries permitted until 2026-07-30 21:03:05.459611441 +0000 UTC m=+357.225573298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert") pod "kserve-controller-manager-d6f4b65b8-clsg8" (UID: "5090fc76-a383-4e9e-8a71-719d87ec90bd") : secret "kserve-webhook-server-cert" not found
Jul 30 21:03:04.962004 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.961973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83e40ff0-3ce8-4f14-a555-364c465aa18a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:04.985899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.985864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4g4\" (UniqueName: \"kubernetes.io/projected/5090fc76-a383-4e9e-8a71-719d87ec90bd-kube-api-access-ws4g4\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:04.986322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:04.986301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxx79\" (UniqueName: \"kubernetes.io/projected/83e40ff0-3ce8-4f14-a555-364c465aa18a-kube-api-access-zxx79\") pod \"llmisvc-controller-manager-777c6bd7d5-876g6\" (UID: \"83e40ff0-3ce8-4f14-a555-364c465aa18a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:05.066171 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.066140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:05.210804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.210632    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"]
Jul 30 21:03:05.214503 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:03:05.214479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod83e40ff0_3ce8_4f14_a555_364c465aa18a.slice/crio-3e3b5055cea519c0d3bae88e8f7c332c68c83ac59fabc1d11c6654c063571cd8 WatchSource:0}: Error finding container 3e3b5055cea519c0d3bae88e8f7c332c68c83ac59fabc1d11c6654c063571cd8: Status 404 returned error can't find the container with id 3e3b5055cea519c0d3bae88e8f7c332c68c83ac59fabc1d11c6654c063571cd8
Jul 30 21:03:05.463709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.463615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:05.466126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.466106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5090fc76-a383-4e9e-8a71-719d87ec90bd-cert\") pod \"kserve-controller-manager-d6f4b65b8-clsg8\" (UID: \"5090fc76-a383-4e9e-8a71-719d87ec90bd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:05.649752 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.649713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:05.811939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.811904    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"]
Jul 30 21:03:05.815876 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:03:05.815847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5090fc76_a383_4e9e_8a71_719d87ec90bd.slice/crio-7543d57d9c1d228439267a1ebc4bdd6c884406e131f1f9b074a76b483728235d WatchSource:0}: Error finding container 7543d57d9c1d228439267a1ebc4bdd6c884406e131f1f9b074a76b483728235d: Status 404 returned error can't find the container with id 7543d57d9c1d228439267a1ebc4bdd6c884406e131f1f9b074a76b483728235d
Jul 30 21:03:05.919805 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:05.919766    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:06.023111 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:06.023036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:06.091560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:06.091527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:06.138075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:06.138038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6" event={"ID":"83e40ff0-3ce8-4f14-a555-364c465aa18a","Type":"ContainerStarted","Data":"3e3b5055cea519c0d3bae88e8f7c332c68c83ac59fabc1d11c6654c063571cd8"}
Jul 30 21:03:06.139339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:06.139308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8" event={"ID":"5090fc76-a383-4e9e-8a71-719d87ec90bd","Type":"ContainerStarted","Data":"7543d57d9c1d228439267a1ebc4bdd6c884406e131f1f9b074a76b483728235d"}
Jul 30 21:03:07.610624 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:07.610585    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:03:07.611107 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:07.610840    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery" containerID="cri-o://444f0479fd41dfcd8bb81749d38d8f0608b9182cb6b84b50297acc6b95841249" gracePeriod=30
Jul 30 21:03:08.149925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:08.149881    2583 generic.go:356] "Generic (PLEG): container finished" podID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerID="444f0479fd41dfcd8bb81749d38d8f0608b9182cb6b84b50297acc6b95841249" exitCode=0
Jul 30 21:03:08.150113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:08.149949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" event={"ID":"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0","Type":"ContainerDied","Data":"444f0479fd41dfcd8bb81749d38d8f0608b9182cb6b84b50297acc6b95841249"}
Jul 30 21:03:08.896738 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:08.896712    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:03:09.095034 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.094995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095034 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095072    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czjdw\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095199    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap\") pod \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\" (UID: \"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0\") "
Jul 30 21:03:09.095723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.095675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:03:09.097738 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.097704    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:03:09.097845 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.097772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:03:09.097893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.097834    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs" (OuterVolumeSpecName: "local-certs") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:03:09.097893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.097840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:03:09.098098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.098062    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw" (OuterVolumeSpecName: "kube-api-access-czjdw") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "kube-api-access-czjdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:03:09.098289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.098275    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts" (OuterVolumeSpecName: "cacerts") pod "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" (UID: "3a5b3984-09e7-4f7d-b0a2-10c3d64003f0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:03:09.105373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.105354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-s4zw5"
Jul 30 21:03:09.154881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.154799    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"
Jul 30 21:03:09.154881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.154798    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp" event={"ID":"3a5b3984-09e7-4f7d-b0a2-10c3d64003f0","Type":"ContainerDied","Data":"c872a6a860e2011d7d53d2260ec7c3fff1d06f46fe1ccd57b2f80470dc9a3281"}
Jul 30 21:03:09.155364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.154929    2583 scope.go:117] "RemoveContainer" containerID="444f0479fd41dfcd8bb81749d38d8f0608b9182cb6b84b50297acc6b95841249"
Jul 30 21:03:09.156980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.156955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8" event={"ID":"5090fc76-a383-4e9e-8a71-719d87ec90bd","Type":"ContainerStarted","Data":"2b70676e7ea9033e5f716c3cc4a1303de8a7aa851d4d3993ef0897e8b0ecaa57"}
Jul 30 21:03:09.157432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.157409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:09.159587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.159242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6" event={"ID":"83e40ff0-3ce8-4f14-a555-364c465aa18a","Type":"ContainerStarted","Data":"5a08fe8b939caaa49b7e9deda9f7529daece3201a150576696ab3ee31c9f8bc5"}
Jul 30 21:03:09.159587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.159411    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:09.176182 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.176129    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8" podStartSLOduration=2.078690623 podStartE2EDuration="5.176111289s" totalImagesPullingTime="3.097420666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:03:09.174543756 +0000 UTC m=+360.940505636" watchObservedRunningTime="2026-07-30 21:03:09.176111289 +0000 UTC m=+360.942073204"
Jul 30 21:03:09.195535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.195484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6" podStartSLOduration=1.496757206 podStartE2EDuration="5.195467935s" totalImagesPullingTime="3.698710729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:03:09.193620777 +0000 UTC m=+360.959582656" watchObservedRunningTime="2026-07-30 21:03:09.195467935 +0000 UTC m=+360.961429858"
Jul 30 21:03:09.196137 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196116    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-dns-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196140    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-kubeconfig\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196149    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-token\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196162    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-cacerts\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196175    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-local-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196184    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjdw\" (UniqueName: \"kubernetes.io/projected/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-kube-api-access-czjdw\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.196213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.196195    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0-istio-csr-ca-configmap\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:03:09.211081 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.211048    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:03:09.217441 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:09.217408    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-q9rhp"]
Jul 30 21:03:10.110392 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:10.110360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-dtlkp"
Jul 30 21:03:10.776288 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:10.776254    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" path="/var/lib/kubelet/pods/3a5b3984-09e7-4f7d-b0a2-10c3d64003f0/volumes"
Jul 30 21:03:40.166559 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:40.166525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-876g6"
Jul 30 21:03:41.173522 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:41.173491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-clsg8"
Jul 30 21:03:43.886161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:43.886129    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:03:43.979060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:03:43.979020    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:02.610789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:02.610754    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:03.399723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:03.399688    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:04.199951 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:04.199913    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:07.200105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.200070    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"]
Jul 30 21:04:07.200488 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.200366    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery"
Jul 30 21:04:07.200488 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.200378    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery"
Jul 30 21:04:07.200488 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.200447    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5b3984-09e7-4f7d-b0a2-10c3d64003f0" containerName="discovery"
Jul 30 21:04:07.202936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.202921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.206652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.206628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 30 21:04:07.210520 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.210490    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 30 21:04:07.210656 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.210600    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6h2kb\""
Jul 30 21:04:07.234922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.234895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"]
Jul 30 21:04:07.293705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.293674    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:07.347525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.347489    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.347525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.347528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjbk\" (UniqueName: \"kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.389904 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.389848    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:07.448821 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.448787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.448821 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.448822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbk\" (UniqueName: \"kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.449156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.449135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.467241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.467171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjbk\" (UniqueName: \"kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk\") pod \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.512961 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.512930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:07.650367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.650331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"]
Jul 30 21:04:07.653915 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:07.653885    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0659586_b51d_4b14_b9e6_107f3a3a8b89.slice/crio-d30fa73c12145d6f68743087a2ad255e35bc46eb795b0c3421009c50061af3da WatchSource:0}: Error finding container d30fa73c12145d6f68743087a2ad255e35bc46eb795b0c3421009c50061af3da: Status 404 returned error can't find the container with id d30fa73c12145d6f68743087a2ad255e35bc46eb795b0c3421009c50061af3da
Jul 30 21:04:07.868661 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:07.868629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:08.378185 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:08.378150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw" event={"ID":"b0659586-b51d-4b14-b9e6-107f3a3a8b89","Type":"ContainerStarted","Data":"d30fa73c12145d6f68743087a2ad255e35bc46eb795b0c3421009c50061af3da"}
Jul 30 21:04:09.122025 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.121990    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"]
Jul 30 21:04:09.126485 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.126458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:09.128556 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.128528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 30 21:04:09.128720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.128604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nz8sz\""
Jul 30 21:04:09.139297 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.139269    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"]
Jul 30 21:04:09.263418 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.263332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zfq\" (UniqueName: \"kubernetes.io/projected/82d57060-bfa9-485e-9045-df0a1e7256d0-kube-api-access-85zfq\") pod \"dns-operator-controller-manager-5985564776-dzj2j\" (UID: \"82d57060-bfa9-485e-9045-df0a1e7256d0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:09.364799 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.364397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zfq\" (UniqueName: \"kubernetes.io/projected/82d57060-bfa9-485e-9045-df0a1e7256d0-kube-api-access-85zfq\") pod \"dns-operator-controller-manager-5985564776-dzj2j\" (UID: \"82d57060-bfa9-485e-9045-df0a1e7256d0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:09.377452 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.377384    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zfq\" (UniqueName: \"kubernetes.io/projected/82d57060-bfa9-485e-9045-df0a1e7256d0-kube-api-access-85zfq\") pod \"dns-operator-controller-manager-5985564776-dzj2j\" (UID: \"82d57060-bfa9-485e-9045-df0a1e7256d0\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:09.440418 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.440373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:09.834320 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.834283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:09.937018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:09.936956    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:10.289669 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:10.289634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"]
Jul 30 21:04:10.293440 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:10.293407    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d57060_bfa9_485e_9045_df0a1e7256d0.slice/crio-c5f45753f0cb793430f49f43bde73ea64836550549e93d071bab2cd34b027614 WatchSource:0}: Error finding container c5f45753f0cb793430f49f43bde73ea64836550549e93d071bab2cd34b027614: Status 404 returned error can't find the container with id c5f45753f0cb793430f49f43bde73ea64836550549e93d071bab2cd34b027614
Jul 30 21:04:10.390965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:10.390875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j" event={"ID":"82d57060-bfa9-485e-9045-df0a1e7256d0","Type":"ContainerStarted","Data":"c5f45753f0cb793430f49f43bde73ea64836550549e93d071bab2cd34b027614"}
Jul 30 21:04:11.759238 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.759205    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zfdnb"]
Jul 30 21:04:11.764000 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.763980    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:11.766027 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.766000    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dv6qh\""
Jul 30 21:04:11.772710 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.772510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zfdnb"]
Jul 30 21:04:11.784124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.784095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmqf\" (UniqueName: \"kubernetes.io/projected/1220c293-f6db-4875-8403-3536ad63e23e-kube-api-access-gvmqf\") pod \"authorino-operator-68d88948d8-zfdnb\" (UID: \"1220c293-f6db-4875-8403-3536ad63e23e\") " pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:11.885061 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.885022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmqf\" (UniqueName: \"kubernetes.io/projected/1220c293-f6db-4875-8403-3536ad63e23e-kube-api-access-gvmqf\") pod \"authorino-operator-68d88948d8-zfdnb\" (UID: \"1220c293-f6db-4875-8403-3536ad63e23e\") " pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:11.894594 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:11.894534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmqf\" (UniqueName: \"kubernetes.io/projected/1220c293-f6db-4875-8403-3536ad63e23e-kube-api-access-gvmqf\") pod \"authorino-operator-68d88948d8-zfdnb\" (UID: \"1220c293-f6db-4875-8403-3536ad63e23e\") " pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:12.077450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:12.077354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:12.405001 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:12.404966    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:12.488933 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:12.488907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-zfdnb"]
Jul 30 21:04:12.662847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:12.662762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:12.751509 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:12.751474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:13.410422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:13.410357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw" event={"ID":"b0659586-b51d-4b14-b9e6-107f3a3a8b89","Type":"ContainerStarted","Data":"53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c"}
Jul 30 21:04:13.410910 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:13.410493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:13.411807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:13.411776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb" event={"ID":"1220c293-f6db-4875-8403-3536ad63e23e","Type":"ContainerStarted","Data":"761bc7e202c765e04026a9914554ecc7a63d628e7530be9710b41586d7a73681"}
Jul 30 21:04:13.427273 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:13.427222    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw" podStartSLOduration=1.6541682309999999 podStartE2EDuration="6.427202367s" totalImagesPullingTime="4.773034136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:13.426554065 +0000 UTC m=+425.192515943" watchObservedRunningTime="2026-07-30 21:04:13.427202367 +0000 UTC m=+425.193164262"
Jul 30 21:04:13.737534 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:13.737492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:14.023550 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.023465    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"]
Jul 30 21:04:14.026847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.026827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:14.028817 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.028791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8ft5t\""
Jul 30 21:04:14.039325 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.039286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"]
Jul 30 21:04:14.099150 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.099113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxpx\" (UniqueName: \"kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx\") pod \"limitador-operator-controller-manager-6c549b6d85-fqz95\" (UID: \"69289a32-cebd-43ca-8032-1f8bcc4eaf88\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:14.199622 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.199556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxpx\" (UniqueName: \"kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx\") pod \"limitador-operator-controller-manager-6c549b6d85-fqz95\" (UID: \"69289a32-cebd-43ca-8032-1f8bcc4eaf88\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:14.208161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.208131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxpx\" (UniqueName: \"kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx\") pod \"limitador-operator-controller-manager-6c549b6d85-fqz95\" (UID: \"69289a32-cebd-43ca-8032-1f8bcc4eaf88\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:14.337697 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.337616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:14.419041 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.418998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j" event={"ID":"82d57060-bfa9-485e-9045-df0a1e7256d0","Type":"ContainerStarted","Data":"ce36404157a1ab62cae4200e5781470494eea26e97d52b813f080692069bad4b"}
Jul 30 21:04:14.435803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.435774    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:14.438623 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.438555    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j" podStartSLOduration=1.835009055 podStartE2EDuration="5.438537197s" totalImagesPullingTime="3.603528142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:14.437472978 +0000 UTC m=+426.203434859" watchObservedRunningTime="2026-07-30 21:04:14.438537197 +0000 UTC m=+426.204499077"
Jul 30 21:04:14.477327 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:14.477298    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"]
Jul 30 21:04:14.480345 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:14.480325    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69289a32_cebd_43ca_8032_1f8bcc4eaf88.slice/crio-0f7b901b590d110c3d96cae132d857a2990e9b659df04a79abe461980b968554 WatchSource:0}: Error finding container 0f7b901b590d110c3d96cae132d857a2990e9b659df04a79abe461980b968554: Status 404 returned error can't find the container with id 0f7b901b590d110c3d96cae132d857a2990e9b659df04a79abe461980b968554
Jul 30 21:04:15.426043 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.426002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb" event={"ID":"1220c293-f6db-4875-8403-3536ad63e23e","Type":"ContainerStarted","Data":"dc530fa69c2508c7fbecb6fe9144fb600cda43468bab6430b396f3b012fc3cdd"}
Jul 30 21:04:15.426465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.426080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:15.427427 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.427402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95" event={"ID":"69289a32-cebd-43ca-8032-1f8bcc4eaf88","Type":"ContainerStarted","Data":"0f7b901b590d110c3d96cae132d857a2990e9b659df04a79abe461980b968554"}
Jul 30 21:04:15.427527 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.427457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:15.441793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.441733    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb" podStartSLOduration=1.981107875 podStartE2EDuration="4.441718818s" totalImagesPullingTime="2.460610943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:15.441062171 +0000 UTC m=+427.207024051" watchObservedRunningTime="2026-07-30 21:04:15.441718818 +0000 UTC m=+427.207680696"
Jul 30 21:04:15.597607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.597543    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:15.791949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:15.791912    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:16.432631 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:16.432593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95" event={"ID":"69289a32-cebd-43ca-8032-1f8bcc4eaf88","Type":"ContainerStarted","Data":"8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a"}
Jul 30 21:04:16.433032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:16.432746    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:16.449939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:16.449894    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95" podStartSLOduration=0.616289816 podStartE2EDuration="2.449880776s" totalImagesPullingTime="1.83359096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:16.448594156 +0000 UTC m=+428.214556029" watchObservedRunningTime="2026-07-30 21:04:16.449880776 +0000 UTC m=+428.215842656"
Jul 30 21:04:17.795907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:17.795877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:17.851423 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:17.851389    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.196508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.196420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.250855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.250822    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.600448 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.600410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.695046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.695014    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.771510 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.771465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:18.906729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:18.906648    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:19.003770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:19.003743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:19.493221 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:19.493187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:19.558273 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:19.558234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:24.421794 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:24.421763    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:24.474763 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:24.474729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:24.575017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:24.574983    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:26.030902 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.030859    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"]
Jul 30 21:04:26.031379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.031148    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" containerName="manager" containerID="cri-o://53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c" gracePeriod=2
Jul 30 21:04:26.037177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.037142    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"]
Jul 30 21:04:26.054461 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.054426    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"]
Jul 30 21:04:26.054995 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.054962    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" containerName="manager" containerID="cri-o://8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a" gracePeriod=2
Jul 30 21:04:26.056359 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.056333    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"]
Jul 30 21:04:26.057310 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.057273    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.059475 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.059193    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.060900 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.060880    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:26.061222 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061207    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" containerName="manager"
Jul 30 21:04:26.061323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061227    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" containerName="manager"
Jul 30 21:04:26.061323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061244    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" containerName="manager"
Jul 30 21:04:26.061323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061253    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" containerName="manager"
Jul 30 21:04:26.061532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061334    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" containerName="manager"
Jul 30 21:04:26.061532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.061349    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" containerName="manager"
Jul 30 21:04:26.065205 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.065187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.066748 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.066722    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.078761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.078732    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"]
Jul 30 21:04:26.082452 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.082427    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:26.082617 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.082558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:26.092558 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.092516    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.093029 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.093005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.093109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.093041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh77f\" (UniqueName: \"kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.093109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.093056    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"]
Jul 30 21:04:26.093109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.093073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77lj\" (UniqueName: \"kubernetes.io/projected/2797eed7-af70-476a-9ce4-1e50a70c0e30-kube-api-access-k77lj\") pod \"limitador-operator-controller-manager-6c549b6d85-hmwjr\" (UID: \"2797eed7-af70-476a-9ce4-1e50a70c0e30\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:26.094790 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.094759    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.096211 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.096182    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.194033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.193990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.194033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.194033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh77f\" (UniqueName: \"kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.194259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.194062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77lj\" (UniqueName: \"kubernetes.io/projected/2797eed7-af70-476a-9ce4-1e50a70c0e30-kube-api-access-k77lj\") pod \"limitador-operator-controller-manager-6c549b6d85-hmwjr\" (UID: \"2797eed7-af70-476a-9ce4-1e50a70c0e30\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:26.194481 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.194456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.203497 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.203469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh77f\" (UniqueName: \"kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f\") pod \"kuadrant-operator-controller-manager-58fc56757c-7t9sx\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.203729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.203708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77lj\" (UniqueName: \"kubernetes.io/projected/2797eed7-af70-476a-9ce4-1e50a70c0e30-kube-api-access-k77lj\") pod \"limitador-operator-controller-manager-6c549b6d85-hmwjr\" (UID: \"2797eed7-af70-476a-9ce4-1e50a70c0e30\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:26.310347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.310324    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.311928 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.311901    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.312402 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.312387    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.313219 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.313197    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.314469 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.314447    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.315736 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.315718    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.395644 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.395613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume\") pod \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") "
Jul 30 21:04:26.395803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.395709    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fxpx\" (UniqueName: \"kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx\") pod \"69289a32-cebd-43ca-8032-1f8bcc4eaf88\" (UID: \"69289a32-cebd-43ca-8032-1f8bcc4eaf88\") "
Jul 30 21:04:26.395803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.395747    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvjbk\" (UniqueName: \"kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk\") pod \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\" (UID: \"b0659586-b51d-4b14-b9e6-107f3a3a8b89\") "
Jul 30 21:04:26.396138 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.396104    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b0659586-b51d-4b14-b9e6-107f3a3a8b89" (UID: "b0659586-b51d-4b14-b9e6-107f3a3a8b89"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:04:26.398011 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.397990    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx" (OuterVolumeSpecName: "kube-api-access-4fxpx") pod "69289a32-cebd-43ca-8032-1f8bcc4eaf88" (UID: "69289a32-cebd-43ca-8032-1f8bcc4eaf88"). InnerVolumeSpecName "kube-api-access-4fxpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:04:26.398106 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.398028    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk" (OuterVolumeSpecName: "kube-api-access-gvjbk") pod "b0659586-b51d-4b14-b9e6-107f3a3a8b89" (UID: "b0659586-b51d-4b14-b9e6-107f3a3a8b89"). InnerVolumeSpecName "kube-api-access-gvjbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:04:26.434824 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.434789    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-dzj2j"
Jul 30 21:04:26.434943 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.434934    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-zfdnb"
Jul 30 21:04:26.447124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.447104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:26.451764 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.451736    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.452820 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.452802    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:26.453245 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.453218    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.454674 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.454654    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.474634 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.474517    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.478218 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.478188    2583 generic.go:356] "Generic (PLEG): container finished" podID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" containerID="8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a" exitCode=0
Jul 30 21:04:26.478378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.478257    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.478378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.478289    2583 scope.go:117] "RemoveContainer" containerID="8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a"
Jul 30 21:04:26.480129 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.480106    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" containerID="53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c" exitCode=0
Jul 30 21:04:26.480217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.480203    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.483789 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.483759    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.485544 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.485386    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.490841 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.490720    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.493242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.491561    2583 scope.go:117] "RemoveContainer" containerID="8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a"
Jul 30 21:04:26.493242 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.491918    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a\": container with ID starting with 8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a not found: ID does not exist" containerID="8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a"
Jul 30 21:04:26.493242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.491951    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a"} err="failed to get container status \"8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a\": rpc error: code = NotFound desc = could not find container \"8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a\": container with ID starting with 8453286f80e9316401ab624b3f2ab32ef387a9c36e36b5abeb29a642864bad8a not found: ID does not exist"
Jul 30 21:04:26.493242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.491975    2583 scope.go:117] "RemoveContainer" containerID="53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c"
Jul 30 21:04:26.493242 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.492167    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.498139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.496954    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxpx\" (UniqueName: \"kubernetes.io/projected/69289a32-cebd-43ca-8032-1f8bcc4eaf88-kube-api-access-4fxpx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:04:26.498139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.496983    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvjbk\" (UniqueName: \"kubernetes.io/projected/b0659586-b51d-4b14-b9e6-107f3a3a8b89-kube-api-access-gvjbk\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:04:26.498139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.496999    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0659586-b51d-4b14-b9e6-107f3a3a8b89-extensions-socket-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:04:26.500932 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.500810    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.502112 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.501967    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.503508 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.503403    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-tpcbw\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-tpcbw"
Jul 30 21:04:26.505186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.504182    2583 scope.go:117] "RemoveContainer" containerID="53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c"
Jul 30 21:04:26.505186 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.504482    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c\": container with ID starting with 53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c not found: ID does not exist" containerID="53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c"
Jul 30 21:04:26.505186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.504512    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c"} err="failed to get container status \"53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c\": rpc error: code = NotFound desc = could not find container \"53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c\": container with ID starting with 53ea60ab57e47843ba8bbaa737e3e16d44c4c7de10a9f0e1382a53ea42d63a8c not found: ID does not exist"
Jul 30 21:04:26.508110 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:26.507872    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-fqz95\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-fqz95"
Jul 30 21:04:26.615697 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.615663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:26.618848 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:26.618822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f988480_5b35_4c5a_ab50_2a84126e1618.slice/crio-9169f0d23c4c9f8f393c48956fdc9bcb252f707afe68466af94af7898b0ac6d7 WatchSource:0}: Error finding container 9169f0d23c4c9f8f393c48956fdc9bcb252f707afe68466af94af7898b0ac6d7: Status 404 returned error can't find the container with id 9169f0d23c4c9f8f393c48956fdc9bcb252f707afe68466af94af7898b0ac6d7
Jul 30 21:04:26.634560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.634540    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"]
Jul 30 21:04:26.637207 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:26.637183    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2797eed7_af70_476a_9ce4_1e50a70c0e30.slice/crio-a5a76a423314056bd7755679ba245a5a703fe122240a15501b1eba1a84bf8197 WatchSource:0}: Error finding container a5a76a423314056bd7755679ba245a5a703fe122240a15501b1eba1a84bf8197: Status 404 returned error can't find the container with id a5a76a423314056bd7755679ba245a5a703fe122240a15501b1eba1a84bf8197
Jul 30 21:04:26.776459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.776418    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69289a32-cebd-43ca-8032-1f8bcc4eaf88" path="/var/lib/kubelet/pods/69289a32-cebd-43ca-8032-1f8bcc4eaf88/volumes"
Jul 30 21:04:26.776785 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:26.776772    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0659586-b51d-4b14-b9e6-107f3a3a8b89" path="/var/lib/kubelet/pods/b0659586-b51d-4b14-b9e6-107f3a3a8b89/volumes"
Jul 30 21:04:27.250810 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.250778    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:04:27.254087 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.254068    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.277115 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.277089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:04:27.303599 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.303502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.303599 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.303545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsr44\" (UniqueName: \"kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.404716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.404676    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.404907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.404735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsr44\" (UniqueName: \"kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.405084 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.405060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.415707 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.415685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsr44\" (UniqueName: \"kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44\") pod \"kuadrant-operator-controller-manager-68cf589589-sqcxf\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.487229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.487195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr" event={"ID":"2797eed7-af70-476a-9ce4-1e50a70c0e30","Type":"ContainerStarted","Data":"050a89afa8819e603768ce944e480580ad374bd629facd7d37fcb8eab9dacda9"}
Jul 30 21:04:27.487409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.487253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr" event={"ID":"2797eed7-af70-476a-9ce4-1e50a70c0e30","Type":"ContainerStarted","Data":"a5a76a423314056bd7755679ba245a5a703fe122240a15501b1eba1a84bf8197"}
Jul 30 21:04:27.487409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.487280    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:27.489916 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.489888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" event={"ID":"8f988480-5b35-4c5a-ab50-2a84126e1618","Type":"ContainerStarted","Data":"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"}
Jul 30 21:04:27.490048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.489939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" event={"ID":"8f988480-5b35-4c5a-ab50-2a84126e1618","Type":"ContainerStarted","Data":"9169f0d23c4c9f8f393c48956fdc9bcb252f707afe68466af94af7898b0ac6d7"}
Jul 30 21:04:27.490048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.489977    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:27.509664 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.509614    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr" podStartSLOduration=1.509599465 podStartE2EDuration="1.509599465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:27.508355336 +0000 UTC m=+439.274317215" watchObservedRunningTime="2026-07-30 21:04:27.509599465 +0000 UTC m=+439.275561337"
Jul 30 21:04:27.537538 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.537489    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" podStartSLOduration=1.5374725599999999 podStartE2EDuration="1.53747256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:27.536211069 +0000 UTC m=+439.302172948" watchObservedRunningTime="2026-07-30 21:04:27.53747256 +0000 UTC m=+439.303434438"
Jul 30 21:04:27.564409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.564346    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:27.709363 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:27.709320    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:04:27.714518 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:27.714489    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa326a8e_f0d3_4f9c_a095_325fa526f0dc.slice/crio-8baad7cbb95460a8be72c9ce57ed2055acb34d4007d96abe9df317243041b10d WatchSource:0}: Error finding container 8baad7cbb95460a8be72c9ce57ed2055acb34d4007d96abe9df317243041b10d: Status 404 returned error can't find the container with id 8baad7cbb95460a8be72c9ce57ed2055acb34d4007d96abe9df317243041b10d
Jul 30 21:04:28.500089 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:28.500043    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" event={"ID":"fa326a8e-f0d3-4f9c-a095-325fa526f0dc","Type":"ContainerStarted","Data":"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"}
Jul 30 21:04:28.500537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:28.500112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" event={"ID":"fa326a8e-f0d3-4f9c-a095-325fa526f0dc","Type":"ContainerStarted","Data":"8baad7cbb95460a8be72c9ce57ed2055acb34d4007d96abe9df317243041b10d"}
Jul 30 21:04:28.500537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:28.500229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:28.521306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:28.521228    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" podStartSLOduration=1.521213596 podStartE2EDuration="1.521213596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:04:28.518823004 +0000 UTC m=+440.284784884" watchObservedRunningTime="2026-07-30 21:04:28.521213596 +0000 UTC m=+440.287175475"
Jul 30 21:04:30.128982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:30.128907    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:30.260755 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:30.260720    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:30.319690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:30.319660    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:31.008109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:31.008081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:31.110118 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:31.110084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:34.017833 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:34.017798    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:34.121689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:34.121657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:38.502897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:38.502868    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:38.503307 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:38.502938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-hmwjr"
Jul 30 21:04:38.955054 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:38.954976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:39.081751 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.081708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:39.167339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.167298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:39.507299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.507268    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:04:39.575042 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.575013    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:39.575238 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.575212    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" podUID="8f988480-5b35-4c5a-ab50-2a84126e1618" containerName="manager" containerID="cri-o://e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be" gracePeriod=10
Jul 30 21:04:39.853432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.853403    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:39.894970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.894937    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh77f\" (UniqueName: \"kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f\") pod \"8f988480-5b35-4c5a-ab50-2a84126e1618\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") "
Jul 30 21:04:39.895109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.895029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume\") pod \"8f988480-5b35-4c5a-ab50-2a84126e1618\" (UID: \"8f988480-5b35-4c5a-ab50-2a84126e1618\") "
Jul 30 21:04:39.895392 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.895366    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8f988480-5b35-4c5a-ab50-2a84126e1618" (UID: "8f988480-5b35-4c5a-ab50-2a84126e1618"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:04:39.897139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.897110    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f" (OuterVolumeSpecName: "kube-api-access-xh77f") pod "8f988480-5b35-4c5a-ab50-2a84126e1618" (UID: "8f988480-5b35-4c5a-ab50-2a84126e1618"). InnerVolumeSpecName "kube-api-access-xh77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:04:39.995809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.995772    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f988480-5b35-4c5a-ab50-2a84126e1618-extensions-socket-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:04:39.995809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:39.995801    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh77f\" (UniqueName: \"kubernetes.io/projected/8f988480-5b35-4c5a-ab50-2a84126e1618-kube-api-access-xh77f\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:04:40.555530 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.555496    2583 generic.go:356] "Generic (PLEG): container finished" podID="8f988480-5b35-4c5a-ab50-2a84126e1618" containerID="e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be" exitCode=0
Jul 30 21:04:40.555959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.555539    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"
Jul 30 21:04:40.555959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.555584    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" event={"ID":"8f988480-5b35-4c5a-ab50-2a84126e1618","Type":"ContainerDied","Data":"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"}
Jul 30 21:04:40.555959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.555629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx" event={"ID":"8f988480-5b35-4c5a-ab50-2a84126e1618","Type":"ContainerDied","Data":"9169f0d23c4c9f8f393c48956fdc9bcb252f707afe68466af94af7898b0ac6d7"}
Jul 30 21:04:40.555959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.555653    2583 scope.go:117] "RemoveContainer" containerID="e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"
Jul 30 21:04:40.564596 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.564578    2583 scope.go:117] "RemoveContainer" containerID="e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"
Jul 30 21:04:40.564833 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:04:40.564816    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be\": container with ID starting with e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be not found: ID does not exist" containerID="e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"
Jul 30 21:04:40.564878 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.564841    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be"} err="failed to get container status \"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be\": rpc error: code = NotFound desc = could not find container \"e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be\": container with ID starting with e891ccf5e0a7e7a929ca426b3a8b85986546556501f9e8740f103f65107a38be not found: ID does not exist"
Jul 30 21:04:40.582641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.582617    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:40.589809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.589786    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-7t9sx"]
Jul 30 21:04:40.776966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:40.776923    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f988480-5b35-4c5a-ab50-2a84126e1618" path="/var/lib/kubelet/pods/8f988480-5b35-4c5a-ab50-2a84126e1618/volumes"
Jul 30 21:04:45.352074 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:45.352039    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:45.407928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:45.407890    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:51.914605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:51.914547    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:52.015909 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:52.015871    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:54.908114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:54.908081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:55.009804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:55.009765    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:55.849579 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:55.849524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:56.088607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:56.088553    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:58.164934 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.164894    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:04:58.165321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.165205    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f988480-5b35-4c5a-ab50-2a84126e1618" containerName="manager"
Jul 30 21:04:58.165321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.165216    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f988480-5b35-4c5a-ab50-2a84126e1618" containerName="manager"
Jul 30 21:04:58.165321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.165273    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f988480-5b35-4c5a-ab50-2a84126e1618" containerName="manager"
Jul 30 21:04:58.167189 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.167164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.168802 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.168773    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:58.169663 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.169644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 30 21:04:58.169741 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.169700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xrbsc\""
Jul 30 21:04:58.176458 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.176430    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:04:58.235533 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.235494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.235766 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.235550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbd4\" (UniqueName: \"kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.267716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.267679    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:04:58.267981 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.267961    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:58.337032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.336994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.337208 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.337046    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbd4\" (UniqueName: \"kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.337666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.337646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.346282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.346243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbd4\" (UniqueName: \"kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4\") pod \"limitador-limitador-859c4d96d6-c7p6b\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.479209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.479169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:04:58.630495 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.630457    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:04:58.633711 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:58.633675    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3985a1d_afc9_4803_bb22_fd9faf6cb0b7.slice/crio-0ecb5181a2214f1edc9aa57628a1d4dca18eb6b4752948405f5d64ce7aca20cd WatchSource:0}: Error finding container 0ecb5181a2214f1edc9aa57628a1d4dca18eb6b4752948405f5d64ce7aca20cd: Status 404 returned error can't find the container with id 0ecb5181a2214f1edc9aa57628a1d4dca18eb6b4752948405f5d64ce7aca20cd
Jul 30 21:04:58.641356 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.641326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" event={"ID":"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7","Type":"ContainerStarted","Data":"0ecb5181a2214f1edc9aa57628a1d4dca18eb6b4752948405f5d64ce7aca20cd"}
Jul 30 21:04:58.652762 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.652728    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:58.969871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.969779    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:04:58.972830 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.972808    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:04:58.975121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.975099    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pzpv2\""
Jul 30 21:04:58.982928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:58.982895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:04:59.009140 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.009106    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:59.042516 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.042475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc26\" (UniqueName: \"kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26\") pod \"authorino-b54df94b4-2wfsb\" (UID: \"b4f4b4c8-211c-41a8-beb9-e37f81fada08\") " pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:04:59.101055 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.101025    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:04:59.104232 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.104199    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:04:59.106590 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.106546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:04:59.112481 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.112449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:04:59.143813 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.143769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc26\" (UniqueName: \"kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26\") pod \"authorino-b54df94b4-2wfsb\" (UID: \"b4f4b4c8-211c-41a8-beb9-e37f81fada08\") " pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:04:59.152808 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.152769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc26\" (UniqueName: \"kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26\") pod \"authorino-b54df94b4-2wfsb\" (UID: \"b4f4b4c8-211c-41a8-beb9-e37f81fada08\") " pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:04:59.244819 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.244766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzxg\" (UniqueName: \"kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg\") pod \"authorino-c548b5d7d-47tww\" (UID: \"80527e52-4e51-44d2-81a2-d1ea789ca139\") " pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:04:59.282691 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.282650    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:04:59.345752 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.345719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzxg\" (UniqueName: \"kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg\") pod \"authorino-c548b5d7d-47tww\" (UID: \"80527e52-4e51-44d2-81a2-d1ea789ca139\") " pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:04:59.353274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.353231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzxg\" (UniqueName: \"kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg\") pod \"authorino-c548b5d7d-47tww\" (UID: \"80527e52-4e51-44d2-81a2-d1ea789ca139\") " pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:04:59.418033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.417997    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:04:59.422437 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.422409    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:04:59.426535 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:59.426506    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4f4b4c8_211c_41a8_beb9_e37f81fada08.slice/crio-cb63e55bd8e602b0d6f8e1d394a9fcb6a1c4c256d55e19c85ea53ae0423ab41a WatchSource:0}: Error finding container cb63e55bd8e602b0d6f8e1d394a9fcb6a1c4c256d55e19c85ea53ae0423ab41a: Status 404 returned error can't find the container with id cb63e55bd8e602b0d6f8e1d394a9fcb6a1c4c256d55e19c85ea53ae0423ab41a
Jul 30 21:04:59.561962 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.561926    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:04:59.564884 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:04:59.564857    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80527e52_4e51_44d2_81a2_d1ea789ca139.slice/crio-74623a3069976fffb86bd088a83d92ea64d7056f10e1bbda952525b71be329f5 WatchSource:0}: Error finding container 74623a3069976fffb86bd088a83d92ea64d7056f10e1bbda952525b71be329f5: Status 404 returned error can't find the container with id 74623a3069976fffb86bd088a83d92ea64d7056f10e1bbda952525b71be329f5
Jul 30 21:04:59.649886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.649847    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-47tww" event={"ID":"80527e52-4e51-44d2-81a2-d1ea789ca139","Type":"ContainerStarted","Data":"74623a3069976fffb86bd088a83d92ea64d7056f10e1bbda952525b71be329f5"}
Jul 30 21:04:59.650993 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:04:59.650963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2wfsb" event={"ID":"b4f4b4c8-211c-41a8-beb9-e37f81fada08","Type":"ContainerStarted","Data":"cb63e55bd8e602b0d6f8e1d394a9fcb6a1c4c256d55e19c85ea53ae0423ab41a"}
Jul 30 21:05:00.111594 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.111544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:00.224165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.224121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:00.509322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.509284    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:00.612189 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.612070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:00.665729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.664991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" event={"ID":"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7","Type":"ContainerStarted","Data":"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"}
Jul 30 21:05:00.665729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.665654    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:05:00.686751 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.685688    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" podStartSLOduration=1.164644014 podStartE2EDuration="2.685665471s" totalImagesPullingTime="1.521021457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:00.683011719 +0000 UTC m=+472.448973600" watchObservedRunningTime="2026-07-30 21:05:00.685665471 +0000 UTC m=+472.451627350"
Jul 30 21:05:00.715049 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:00.714995    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:01.003537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:01.003496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:02.677790 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:02.677750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-47tww" event={"ID":"80527e52-4e51-44d2-81a2-d1ea789ca139","Type":"ContainerStarted","Data":"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"}
Jul 30 21:05:02.679461 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:02.679433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2wfsb" event={"ID":"b4f4b4c8-211c-41a8-beb9-e37f81fada08","Type":"ContainerStarted","Data":"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"}
Jul 30 21:05:02.695401 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:02.695342    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-47tww" podStartSLOduration=1.464121008 podStartE2EDuration="3.695320363s" totalImagesPullingTime="2.231199355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:02.693292322 +0000 UTC m=+474.459254202" watchObservedRunningTime="2026-07-30 21:05:02.695320363 +0000 UTC m=+474.461282343"
Jul 30 21:05:02.709943 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:02.709873    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-2wfsb" podStartSLOduration=2.351369118 podStartE2EDuration="4.709846391s" totalImagesPullingTime="2.358477273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:02.708768945 +0000 UTC m=+474.474730824" watchObservedRunningTime="2026-07-30 21:05:02.709846391 +0000 UTC m=+474.475808270"
Jul 30 21:05:02.735435 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:02.735398    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:05:04.689244 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:04.689206    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-2wfsb" podUID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" containerName="authorino" containerID="cri-o://7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968" gracePeriod=30
Jul 30 21:05:04.935401 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:04.935370    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:05:05.090728 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.090694    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gc26\" (UniqueName: \"kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26\") pod \"b4f4b4c8-211c-41a8-beb9-e37f81fada08\" (UID: \"b4f4b4c8-211c-41a8-beb9-e37f81fada08\") "
Jul 30 21:05:05.092906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.092875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26" (OuterVolumeSpecName: "kube-api-access-4gc26") pod "b4f4b4c8-211c-41a8-beb9-e37f81fada08" (UID: "b4f4b4c8-211c-41a8-beb9-e37f81fada08"). InnerVolumeSpecName "kube-api-access-4gc26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:05.191961 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.191922    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gc26\" (UniqueName: \"kubernetes.io/projected/b4f4b4c8-211c-41a8-beb9-e37f81fada08-kube-api-access-4gc26\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:05.695289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.695253    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" containerID="7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968" exitCode=0
Jul 30 21:05:05.695690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.695302    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-2wfsb"
Jul 30 21:05:05.695690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.695343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2wfsb" event={"ID":"b4f4b4c8-211c-41a8-beb9-e37f81fada08","Type":"ContainerDied","Data":"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"}
Jul 30 21:05:05.695690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.695386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-2wfsb" event={"ID":"b4f4b4c8-211c-41a8-beb9-e37f81fada08","Type":"ContainerDied","Data":"cb63e55bd8e602b0d6f8e1d394a9fcb6a1c4c256d55e19c85ea53ae0423ab41a"}
Jul 30 21:05:05.695690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.695404    2583 scope.go:117] "RemoveContainer" containerID="7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"
Jul 30 21:05:05.703965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.703950    2583 scope.go:117] "RemoveContainer" containerID="7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"
Jul 30 21:05:05.704250 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:05.704233    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968\": container with ID starting with 7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968 not found: ID does not exist" containerID="7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"
Jul 30 21:05:05.704292 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.704258    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968"} err="failed to get container status \"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968\": rpc error: code = NotFound desc = could not find container \"7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968\": container with ID starting with 7256dfbb886e660e68c4664e5e0e30642d0893d2744312039052733d340f7968 not found: ID does not exist"
Jul 30 21:05:05.715313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.715289    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:05:05.718974 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:05.718956    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-2wfsb"]
Jul 30 21:05:06.776559 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:06.776528    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" path="/var/lib/kubelet/pods/b4f4b4c8-211c-41a8-beb9-e37f81fada08/volumes"
Jul 30 21:05:07.179943 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:07.179862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:07.488400 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:07.488370    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:11.672895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:11.672854    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:05:13.472884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:13.472846    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:05:13.473298 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:13.473021    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" podUID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" containerName="limitador" containerID="cri-o://aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0" gracePeriod=30
Jul 30 21:05:13.730591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:13.730546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:13.832047 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:13.832020    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:14.023945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.023920    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:05:14.160321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.160288    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxbd4\" (UniqueName: \"kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4\") pod \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") "
Jul 30 21:05:14.160321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.160325    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file\") pod \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\" (UID: \"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7\") "
Jul 30 21:05:14.160756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.160732    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file" (OuterVolumeSpecName: "config-file") pod "b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" (UID: "b3985a1d-afc9-4803-bb22-fd9faf6cb0b7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:05:14.162547 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.162526    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4" (OuterVolumeSpecName: "kube-api-access-wxbd4") pod "b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" (UID: "b3985a1d-afc9-4803-bb22-fd9faf6cb0b7"). InnerVolumeSpecName "kube-api-access-wxbd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:14.261797 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.261714    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbd4\" (UniqueName: \"kubernetes.io/projected/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-kube-api-access-wxbd4\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:14.261797 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.261743    2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7-config-file\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:14.738102 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.738068    2583 generic.go:356] "Generic (PLEG): container finished" podID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" containerID="aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0" exitCode=0
Jul 30 21:05:14.738590 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.738115    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"
Jul 30 21:05:14.738590 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.738167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" event={"ID":"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7","Type":"ContainerDied","Data":"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"}
Jul 30 21:05:14.738590 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.738233    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-c7p6b" event={"ID":"b3985a1d-afc9-4803-bb22-fd9faf6cb0b7","Type":"ContainerDied","Data":"0ecb5181a2214f1edc9aa57628a1d4dca18eb6b4752948405f5d64ce7aca20cd"}
Jul 30 21:05:14.738590 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.738269    2583 scope.go:117] "RemoveContainer" containerID="aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"
Jul 30 21:05:14.746782 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.746768    2583 scope.go:117] "RemoveContainer" containerID="aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"
Jul 30 21:05:14.747042 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:14.747027    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0\": container with ID starting with aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0 not found: ID does not exist" containerID="aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"
Jul 30 21:05:14.747080 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.747051    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0"} err="failed to get container status \"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0\": rpc error: code = NotFound desc = could not find container \"aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0\": container with ID starting with aaa3ebb6a14d0e3190f9428d60ae976798ff4604c42fa3026b12049c3a33f7d0 not found: ID does not exist"
Jul 30 21:05:14.756844 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.756823    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:05:14.758812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.758792    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-c7p6b"]
Jul 30 21:05:14.775891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:14.775872    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" path="/var/lib/kubelet/pods/b3985a1d-afc9-4803-bb22-fd9faf6cb0b7/volumes"
Jul 30 21:05:16.727671 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:16.727637    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:16.828879 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:16.828842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:19.326057 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.326022    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:19.457317 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.457277    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:19.703339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703267    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"]
Jul 30 21:05:19.703588 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703557    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" containerName="limitador"
Jul 30 21:05:19.703643 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703589    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" containerName="limitador"
Jul 30 21:05:19.703643 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703609    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" containerName="authorino"
Jul 30 21:05:19.703643 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703614    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" containerName="authorino"
Jul 30 21:05:19.703773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703660    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f4b4c8-211c-41a8-beb9-e37f81fada08" containerName="authorino"
Jul 30 21:05:19.703773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.703670    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3985a1d-afc9-4803-bb22-fd9faf6cb0b7" containerName="limitador"
Jul 30 21:05:19.708014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.707996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.709922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.709903    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 30 21:05:19.710022 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.709904    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j4qkn\""
Jul 30 21:05:19.710284 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.710266    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 30 21:05:19.710347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.710320    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 30 21:05:19.716155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.716136    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"]
Jul 30 21:05:19.731549 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.731519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:19.803177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.803150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19e75ea8-7888-4b10-ad02-2b78c142e95a-data\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.803300 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.803206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jbt\" (UniqueName: \"kubernetes.io/projected/19e75ea8-7888-4b10-ad02-2b78c142e95a-kube-api-access-p6jbt\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.904398 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.904369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19e75ea8-7888-4b10-ad02-2b78c142e95a-data\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.904536 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.904421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jbt\" (UniqueName: \"kubernetes.io/projected/19e75ea8-7888-4b10-ad02-2b78c142e95a-kube-api-access-p6jbt\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.904753 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.904734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/19e75ea8-7888-4b10-ad02-2b78c142e95a-data\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:19.912744 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:19.912713    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jbt\" (UniqueName: \"kubernetes.io/projected/19e75ea8-7888-4b10-ad02-2b78c142e95a-kube-api-access-p6jbt\") pod \"postgres-69fb4594cf-kgft4\" (UID: \"19e75ea8-7888-4b10-ad02-2b78c142e95a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:20.018991 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:20.018945    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:20.150013 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:20.149982    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"]
Jul 30 21:05:20.154069 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:05:20.154043    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e75ea8_7888_4b10_ad02_2b78c142e95a.slice/crio-bb829102e77dc94e29371f155a2ed8d79934477f4355b5d6722ffa92af632854 WatchSource:0}: Error finding container bb829102e77dc94e29371f155a2ed8d79934477f4355b5d6722ffa92af632854: Status 404 returned error can't find the container with id bb829102e77dc94e29371f155a2ed8d79934477f4355b5d6722ffa92af632854
Jul 30 21:05:20.765610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:20.765549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4" event={"ID":"19e75ea8-7888-4b10-ad02-2b78c142e95a","Type":"ContainerStarted","Data":"bb829102e77dc94e29371f155a2ed8d79934477f4355b5d6722ffa92af632854"}
Jul 30 21:05:25.800498 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:25.800457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4" event={"ID":"19e75ea8-7888-4b10-ad02-2b78c142e95a","Type":"ContainerStarted","Data":"cfb57e423027af19791a35d7fc3990b5b8cce6ccc1a3272635608b5221205bbf"}
Jul 30 21:05:25.800969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:25.800620    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:25.815396 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:25.815354    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4" podStartSLOduration=1.6471076999999998 podStartE2EDuration="6.815336827s" totalImagesPullingTime="5.168229127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:25.814442465 +0000 UTC m=+497.580404344" watchObservedRunningTime="2026-07-30 21:05:25.815336827 +0000 UTC m=+497.581298706"
Jul 30 21:05:31.833234 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:31.833205    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kgft4"
Jul 30 21:05:32.131197 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.131085    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:32.352702 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.352661    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8vp9v"]
Jul 30 21:05:32.356026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.356002    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.364780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.364753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8vp9v"]
Jul 30 21:05:32.374197 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.374168    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:32.406439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.406378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz87q\" (UniqueName: \"kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q\") pod \"authorino-5dc7ccb5f-8vp9v\" (UID: \"edfce2c3-d39e-48f7-b22b-03970a54f93f\") " pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.410594 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.410539    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8vp9v"]
Jul 30 21:05:32.410810 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:32.410787    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sz87q], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v" podUID="edfce2c3-d39e-48f7-b22b-03970a54f93f"
Jul 30 21:05:32.434966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.434935    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-jjprr"]
Jul 30 21:05:32.438134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.438112    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.440081 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.440064    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 30 21:05:32.446027 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.446000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-jjprr"]
Jul 30 21:05:32.507636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.507600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz87q\" (UniqueName: \"kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q\") pod \"authorino-5dc7ccb5f-8vp9v\" (UID: \"edfce2c3-d39e-48f7-b22b-03970a54f93f\") " pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.507803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.507698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhr66\" (UniqueName: \"kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.507803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.507737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.515006 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.514984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz87q\" (UniqueName: \"kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q\") pod \"authorino-5dc7ccb5f-8vp9v\" (UID: \"edfce2c3-d39e-48f7-b22b-03970a54f93f\") " pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.578149 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.578114    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-jjprr"]
Jul 30 21:05:32.578381 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:32.578358    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rhr66 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-jjprr" podUID="7c571d39-e999-400f-9a8e-e4493d0fb56c"
Jul 30 21:05:32.605552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.605526    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:05:32.608241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.608214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhr66\" (UniqueName: \"kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.608370 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.608251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.608859 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.608843    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.610782 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.610764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.616379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.616353    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:05:32.618742 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.618694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhr66\" (UniqueName: \"kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66\") pod \"authorino-687bd45585-jjprr\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") " pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.709394 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.709314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.709394 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.709350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxw5\" (UniqueName: \"kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.810416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.810384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.810416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.810417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxw5\" (UniqueName: \"kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.812969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.812950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.818258 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.818235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxw5\" (UniqueName: \"kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5\") pod \"authorino-6cb899984c-q2hts\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") " pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:32.829073 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.829051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.829171 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.829051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.835998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.835978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:32.838632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.838616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:32.910743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.910718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert\") pod \"7c571d39-e999-400f-9a8e-e4493d0fb56c\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") "
Jul 30 21:05:32.910876 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.910769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz87q\" (UniqueName: \"kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q\") pod \"edfce2c3-d39e-48f7-b22b-03970a54f93f\" (UID: \"edfce2c3-d39e-48f7-b22b-03970a54f93f\") "
Jul 30 21:05:32.910876 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.910839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhr66\" (UniqueName: \"kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66\") pod \"7c571d39-e999-400f-9a8e-e4493d0fb56c\" (UID: \"7c571d39-e999-400f-9a8e-e4493d0fb56c\") "
Jul 30 21:05:32.913082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.913049    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7c571d39-e999-400f-9a8e-e4493d0fb56c" (UID: "7c571d39-e999-400f-9a8e-e4493d0fb56c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:05:32.913082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.913058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q" (OuterVolumeSpecName: "kube-api-access-sz87q") pod "edfce2c3-d39e-48f7-b22b-03970a54f93f" (UID: "edfce2c3-d39e-48f7-b22b-03970a54f93f"). InnerVolumeSpecName "kube-api-access-sz87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:32.913213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.913058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66" (OuterVolumeSpecName: "kube-api-access-rhr66") pod "7c571d39-e999-400f-9a8e-e4493d0fb56c" (UID: "7c571d39-e999-400f-9a8e-e4493d0fb56c"). InnerVolumeSpecName "kube-api-access-rhr66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:32.936232 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:32.936209    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:05:33.011855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.011830    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhr66\" (UniqueName: \"kubernetes.io/projected/7c571d39-e999-400f-9a8e-e4493d0fb56c-kube-api-access-rhr66\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:33.011855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.011855    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c571d39-e999-400f-9a8e-e4493d0fb56c-tls-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:33.012030 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.011866    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz87q\" (UniqueName: \"kubernetes.io/projected/edfce2c3-d39e-48f7-b22b-03970a54f93f-kube-api-access-sz87q\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:33.063362 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.063327    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:05:33.066327 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:05:33.066302    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ee43205_c62a_4bdb_b85e_86bc653e2c82.slice/crio-eed46e6ee94c526e5f54f1c30f0d6c177a3406c433e3bc6d06e243aa78e1c104 WatchSource:0}: Error finding container eed46e6ee94c526e5f54f1c30f0d6c177a3406c433e3bc6d06e243aa78e1c104: Status 404 returned error can't find the container with id eed46e6ee94c526e5f54f1c30f0d6c177a3406c433e3bc6d06e243aa78e1c104
Jul 30 21:05:33.834916 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.834831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-8vp9v"
Jul 30 21:05:33.834916 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.834865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jjprr"
Jul 30 21:05:33.835107 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.834831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cb899984c-q2hts" event={"ID":"6ee43205-c62a-4bdb-b85e-86bc653e2c82","Type":"ContainerStarted","Data":"eed46e6ee94c526e5f54f1c30f0d6c177a3406c433e3bc6d06e243aa78e1c104"}
Jul 30 21:05:33.867846 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.866604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-jjprr"]
Jul 30 21:05:33.870967 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.870933    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-jjprr"]
Jul 30 21:05:33.891999 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.891968    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8vp9v"]
Jul 30 21:05:33.895132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:33.895103    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-8vp9v"]
Jul 30 21:05:34.776960 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.776927    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c571d39-e999-400f-9a8e-e4493d0fb56c" path="/var/lib/kubelet/pods/7c571d39-e999-400f-9a8e-e4493d0fb56c/volumes"
Jul 30 21:05:34.777159 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.777137    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edfce2c3-d39e-48f7-b22b-03970a54f93f" path="/var/lib/kubelet/pods/edfce2c3-d39e-48f7-b22b-03970a54f93f/volumes"
Jul 30 21:05:34.840290 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.840257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cb899984c-q2hts" event={"ID":"6ee43205-c62a-4bdb-b85e-86bc653e2c82","Type":"ContainerStarted","Data":"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"}
Jul 30 21:05:34.855795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.855737    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cb899984c-q2hts" podStartSLOduration=1.5760219229999999 podStartE2EDuration="2.855719556s" totalImagesPullingTime="1.279697633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:34.855099538 +0000 UTC m=+506.621061417" watchObservedRunningTime="2026-07-30 21:05:34.855719556 +0000 UTC m=+506.621681436"
Jul 30 21:05:34.879789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.879752    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:05:34.880194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:34.879955    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-47tww" podUID="80527e52-4e51-44d2-81a2-d1ea789ca139" containerName="authorino" containerID="cri-o://9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49" gracePeriod=30
Jul 30 21:05:35.190108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.190079    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:05:35.230585 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.230544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmzxg\" (UniqueName: \"kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg\") pod \"80527e52-4e51-44d2-81a2-d1ea789ca139\" (UID: \"80527e52-4e51-44d2-81a2-d1ea789ca139\") "
Jul 30 21:05:35.232959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.232936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg" (OuterVolumeSpecName: "kube-api-access-wmzxg") pod "80527e52-4e51-44d2-81a2-d1ea789ca139" (UID: "80527e52-4e51-44d2-81a2-d1ea789ca139"). InnerVolumeSpecName "kube-api-access-wmzxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:35.331336 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.331261    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzxg\" (UniqueName: \"kubernetes.io/projected/80527e52-4e51-44d2-81a2-d1ea789ca139-kube-api-access-wmzxg\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:35.846144 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.846113    2583 generic.go:356] "Generic (PLEG): container finished" podID="80527e52-4e51-44d2-81a2-d1ea789ca139" containerID="9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49" exitCode=0
Jul 30 21:05:35.846324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.846150    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-47tww"
Jul 30 21:05:35.846324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.846200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-47tww" event={"ID":"80527e52-4e51-44d2-81a2-d1ea789ca139","Type":"ContainerDied","Data":"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"}
Jul 30 21:05:35.846324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.846239    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-47tww" event={"ID":"80527e52-4e51-44d2-81a2-d1ea789ca139","Type":"ContainerDied","Data":"74623a3069976fffb86bd088a83d92ea64d7056f10e1bbda952525b71be329f5"}
Jul 30 21:05:35.846324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.846262    2583 scope.go:117] "RemoveContainer" containerID="9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"
Jul 30 21:05:35.854580 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.854547    2583 scope.go:117] "RemoveContainer" containerID="9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"
Jul 30 21:05:35.854846 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:35.854825    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49\": container with ID starting with 9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49 not found: ID does not exist" containerID="9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"
Jul 30 21:05:35.854940 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.854851    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49"} err="failed to get container status \"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49\": rpc error: code = NotFound desc = could not find container \"9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49\": container with ID starting with 9c3233537dae753dd39f75ed59150e2a5ca30f2daf73da36a6196b99dcbdbe49 not found: ID does not exist"
Jul 30 21:05:35.864824 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.864803    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:05:35.868656 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:35.868631    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-47tww"]
Jul 30 21:05:36.777068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:36.777033    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80527e52-4e51-44d2-81a2-d1ea789ca139" path="/var/lib/kubelet/pods/80527e52-4e51-44d2-81a2-d1ea789ca139/volumes"
Jul 30 21:05:39.326040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.326002    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:39.334120 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.334094    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:39.334502 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.334482    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80527e52-4e51-44d2-81a2-d1ea789ca139" containerName="authorino"
Jul 30 21:05:39.334610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.334504    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="80527e52-4e51-44d2-81a2-d1ea789ca139" containerName="authorino"
Jul 30 21:05:39.334610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.334583    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="80527e52-4e51-44d2-81a2-d1ea789ca139" containerName="authorino"
Jul 30 21:05:39.338891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.338870    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.340845 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.340824    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 21:05:39.340976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.340824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 21:05:39.340976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.340913    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67mks\""
Jul 30 21:05:39.349414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.349379    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:39.363963 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.363934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkh5\" (UniqueName: \"kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.364114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.363972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.422906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.422875    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:39.465227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.465191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkh5\" (UniqueName: \"kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.465227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.465230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.465431 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:39.465345    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 30 21:05:39.465431 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:39.465413    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert podName:c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c nodeName:}" failed. No retries permitted until 2026-07-30 21:05:39.965392037 +0000 UTC m=+511.731353914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert") pod "maas-controller-7b56bc7996-wtw9b" (UID: "c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c") : secret "maas-controller-webhook-cert" not found
Jul 30 21:05:39.475078 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.475050    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkh5\" (UniqueName: \"kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.488891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.488859    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:39.492750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.492722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:05:39.495017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.494998    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.505007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.504973    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:05:39.547660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.547627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:39.565755 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.565730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.565837 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.565762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk545\" (UniqueName: \"kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.667184 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.667092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.667184 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.667141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk545\" (UniqueName: \"kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.669680 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.669648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.674940 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.674918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk545\" (UniqueName: \"kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545\") pod \"maas-controller-7d7c48cdc9-rbtp7\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") " pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.805356 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.805319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:39.949412 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.949367    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:05:39.952119 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:05:39.952087    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f81515_b87b_4fb1_b85a_160c8505a61f.slice/crio-17120efe2f58280a0b657f146ea4cfba6dc6fb60a85c486b00eaef578e0bf7ef WatchSource:0}: Error finding container 17120efe2f58280a0b657f146ea4cfba6dc6fb60a85c486b00eaef578e0bf7ef: Status 404 returned error can't find the container with id 17120efe2f58280a0b657f146ea4cfba6dc6fb60a85c486b00eaef578e0bf7ef
Jul 30 21:05:39.969743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.969718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:39.972161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:39.972135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") pod \"maas-controller-7b56bc7996-wtw9b\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") " pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:40.251291 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:40.251258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:40.383625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:40.383546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:40.386381 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:05:40.386347    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d76cba_41a7_4f7f_bd3d_039bc8d9c64c.slice/crio-5a7108b4dd4a3b7c5d7634d745ab6554c7cf2981ac3f31b905685cb85ddd3067 WatchSource:0}: Error finding container 5a7108b4dd4a3b7c5d7634d745ab6554c7cf2981ac3f31b905685cb85ddd3067: Status 404 returned error can't find the container with id 5a7108b4dd4a3b7c5d7634d745ab6554c7cf2981ac3f31b905685cb85ddd3067
Jul 30 21:05:40.871906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:40.871866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" event={"ID":"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c","Type":"ContainerStarted","Data":"5a7108b4dd4a3b7c5d7634d745ab6554c7cf2981ac3f31b905685cb85ddd3067"}
Jul 30 21:05:40.873295 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:40.873261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" event={"ID":"d1f81515-b87b-4fb1-b85a-160c8505a61f","Type":"ContainerStarted","Data":"17120efe2f58280a0b657f146ea4cfba6dc6fb60a85c486b00eaef578e0bf7ef"}
Jul 30 21:05:42.884671 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.884528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" event={"ID":"d1f81515-b87b-4fb1-b85a-160c8505a61f","Type":"ContainerStarted","Data":"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"}
Jul 30 21:05:42.884671 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.884634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:42.886215 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.886186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" event={"ID":"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c","Type":"ContainerStarted","Data":"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"}
Jul 30 21:05:42.886368 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.886290    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:42.901369 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.901338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:42.928625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.928560    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" podStartSLOduration=1.186725441 podStartE2EDuration="3.928545985s" totalImagesPullingTime="2.741820544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:42.926341303 +0000 UTC m=+514.692303187" watchObservedRunningTime="2026-07-30 21:05:42.928545985 +0000 UTC m=+514.694507864"
Jul 30 21:05:42.964764 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:42.964667    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" podStartSLOduration=1.651014875 podStartE2EDuration="3.964652121s" totalImagesPullingTime="2.313637246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:05:42.963847798 +0000 UTC m=+514.729809688" watchObservedRunningTime="2026-07-30 21:05:42.964652121 +0000 UTC m=+514.730614000"
Jul 30 21:05:43.046890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:43.046849    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:05:53.893351 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:53.893316    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:05:53.893807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:53.893387    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:53.951289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:53.951251    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:53.951545 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:53.951474    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" podUID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" containerName="manager" containerID="cri-o://4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05" gracePeriod=30
Jul 30 21:05:54.197608 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.197559    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:54.292092 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.292058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") pod \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") "
Jul 30 21:05:54.292254 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.292134    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbkh5\" (UniqueName: \"kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5\") pod \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\" (UID: \"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c\") "
Jul 30 21:05:54.294409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.294372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" (UID: "c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:05:54.294409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.294372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5" (OuterVolumeSpecName: "kube-api-access-lbkh5") pod "c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" (UID: "c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c"). InnerVolumeSpecName "kube-api-access-lbkh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:05:54.393007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.392968    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-webhook-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:54.393007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.393002    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkh5\" (UniqueName: \"kubernetes.io/projected/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c-kube-api-access-lbkh5\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:05:54.938018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.937985    2583 generic.go:356] "Generic (PLEG): container finished" podID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" containerID="4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05" exitCode=0
Jul 30 21:05:54.938414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.938042    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b56bc7996-wtw9b"
Jul 30 21:05:54.938414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.938072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" event={"ID":"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c","Type":"ContainerDied","Data":"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"}
Jul 30 21:05:54.938414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.938149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b56bc7996-wtw9b" event={"ID":"c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c","Type":"ContainerDied","Data":"5a7108b4dd4a3b7c5d7634d745ab6554c7cf2981ac3f31b905685cb85ddd3067"}
Jul 30 21:05:54.938414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.938170    2583 scope.go:117] "RemoveContainer" containerID="4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"
Jul 30 21:05:54.946535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.946517    2583 scope.go:117] "RemoveContainer" containerID="4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"
Jul 30 21:05:54.946814 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:05:54.946798    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05\": container with ID starting with 4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05 not found: ID does not exist" containerID="4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"
Jul 30 21:05:54.946872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.946831    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05"} err="failed to get container status \"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05\": rpc error: code = NotFound desc = could not find container \"4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05\": container with ID starting with 4d23cdb56799881503529a82b680f80a3abbb8fb646fab9771c58596af11df05 not found: ID does not exist"
Jul 30 21:05:54.952386 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.952364    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:54.956798 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:54.956776    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b56bc7996-wtw9b"]
Jul 30 21:05:56.777042 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:56.777012    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" path="/var/lib/kubelet/pods/c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c/volumes"
Jul 30 21:05:59.853698 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:05:59.853612    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:14.584650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:14.584524    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:14.685105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:14.685070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:16.242893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.242853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:16.358107 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.358070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:16.866818 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.866785    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-stmd5"]
Jul 30 21:06:16.867113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.867102    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" containerName="manager"
Jul 30 21:06:16.867158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.867115    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" containerName="manager"
Jul 30 21:06:16.867191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.867163    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d76cba-41a7-4f7f-bd3d-039bc8d9c64c" containerName="manager"
Jul 30 21:06:16.875451 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.875426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:16.878800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.878779    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 30 21:06:16.878928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.878803    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rfvqg\""
Jul 30 21:06:16.884835 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.884814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-stmd5"]
Jul 30 21:06:16.926740 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.926701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:16.976933 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.976901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpt8\" (UniqueName: \"kubernetes.io/projected/82fa765e-1635-4898-afdb-718f76ff6b11-kube-api-access-blpt8\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:16.977107 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:16.976954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fa765e-1635-4898-afdb-718f76ff6b11-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.027347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.027311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:17.078129 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.078095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fa765e-1635-4898-afdb-718f76ff6b11-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.078322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.078164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpt8\" (UniqueName: \"kubernetes.io/projected/82fa765e-1635-4898-afdb-718f76ff6b11-kube-api-access-blpt8\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.078952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.078924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82fa765e-1635-4898-afdb-718f76ff6b11-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.085725 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.085700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpt8\" (UniqueName: \"kubernetes.io/projected/82fa765e-1635-4898-afdb-718f76ff6b11-kube-api-access-blpt8\") pod \"payload-processing-5cc6cd444b-stmd5\" (UID: \"82fa765e-1635-4898-afdb-718f76ff6b11\") " pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.187315 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.187226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:17.321531 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:17.321497    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-stmd5"]
Jul 30 21:06:17.324393 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:06:17.324362    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa765e_1635_4898_afdb_718f76ff6b11.slice/crio-3e236c19ebc2538a85d16e38c822fb5b770858143b6c16b1784da5bca1c47872 WatchSource:0}: Error finding container 3e236c19ebc2538a85d16e38c822fb5b770858143b6c16b1784da5bca1c47872: Status 404 returned error can't find the container with id 3e236c19ebc2538a85d16e38c822fb5b770858143b6c16b1784da5bca1c47872
Jul 30 21:06:18.047942 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:18.047901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5" event={"ID":"82fa765e-1635-4898-afdb-718f76ff6b11","Type":"ContainerStarted","Data":"3e236c19ebc2538a85d16e38c822fb5b770858143b6c16b1784da5bca1c47872"}
Jul 30 21:06:20.061008 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:20.060968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5" event={"ID":"82fa765e-1635-4898-afdb-718f76ff6b11","Type":"ContainerStarted","Data":"7a6d8539d998cdcfbb99c35730e0a6f6fddc45e5de8d1394f0f21a37ca309d0d"}
Jul 30 21:06:20.061455 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:20.061102    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:20.077810 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:20.077763    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5" podStartSLOduration=2.364867196 podStartE2EDuration="4.077747871s" totalImagesPullingTime="1.712880675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:20.07555231 +0000 UTC m=+551.841514189" watchObservedRunningTime="2026-07-30 21:06:20.077747871 +0000 UTC m=+551.843709802"
Jul 30 21:06:28.528087 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:28.528048    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:28.633070 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:28.633032    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:31.068704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:31.068675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-stmd5"
Jul 30 21:06:40.483423 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:40.483385    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:40.539985 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:40.539947    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:53.862748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.862709    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"]
Jul 30 21:06:53.869380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.869360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.871372 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.871355    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 30 21:06:53.871937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.871908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9gzdm\""
Jul 30 21:06:53.872046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.871959    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 30 21:06:53.872046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.871959    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 30 21:06:53.876126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.876103    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"]
Jul 30 21:06:53.893424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.893402    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:06:53.960514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.960713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.960713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7mr\" (UniqueName: \"kubernetes.io/projected/228a93d3-ca8a-4f74-9494-4ac0c2808531-kube-api-access-qj7mr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.960713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/228a93d3-ca8a-4f74-9494-4ac0c2808531-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.960713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:53.960884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:53.960733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.061802 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7mr\" (UniqueName: \"kubernetes.io/projected/228a93d3-ca8a-4f74-9494-4ac0c2808531-kube-api-access-qj7mr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.061904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/228a93d3-ca8a-4f74-9494-4ac0c2808531-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.062178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.062258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.062354 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.062288    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.064213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.064193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/228a93d3-ca8a-4f74-9494-4ac0c2808531-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.064482 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.064465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/228a93d3-ca8a-4f74-9494-4ac0c2808531-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.069797 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.069769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7mr\" (UniqueName: \"kubernetes.io/projected/228a93d3-ca8a-4f74-9494-4ac0c2808531-kube-api-access-qj7mr\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt\" (UID: \"228a93d3-ca8a-4f74-9494-4ac0c2808531\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.180450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.180370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:06:54.316379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:54.316331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"]
Jul 30 21:06:54.320787 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:06:54.320753    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228a93d3_ca8a_4f74_9494_4ac0c2808531.slice/crio-a3ebfe064b9811d8590ec669fd32c9f16e8165855e31e1edc69b491f55f05f6a WatchSource:0}: Error finding container a3ebfe064b9811d8590ec669fd32c9f16e8165855e31e1edc69b491f55f05f6a: Status 404 returned error can't find the container with id a3ebfe064b9811d8590ec669fd32c9f16e8165855e31e1edc69b491f55f05f6a
Jul 30 21:06:55.216645 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:55.216603    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt" event={"ID":"228a93d3-ca8a-4f74-9494-4ac0c2808531","Type":"ContainerStarted","Data":"a3ebfe064b9811d8590ec669fd32c9f16e8165855e31e1edc69b491f55f05f6a"}
Jul 30 21:06:56.222888 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:56.222853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt" event={"ID":"228a93d3-ca8a-4f74-9494-4ac0c2808531","Type":"ContainerStarted","Data":"1184e3b6296fee83c5ea6b85d91f5b7d7ee5bddf3aa80e1f509a102bba159370"}
Jul 30 21:06:56.239691 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:06:56.239641    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt" podStartSLOduration=1.94072599 podStartE2EDuration="3.239626384s" totalImagesPullingTime="1.298900394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:06:56.237894761 +0000 UTC m=+588.003856640" watchObservedRunningTime="2026-07-30 21:06:56.239626384 +0000 UTC m=+588.005588263"
Jul 30 21:07:04.181259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:04.181215    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:07:04.181683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:04.181331    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:07:04.193793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:04.193770    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:07:04.267376 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:04.267347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt"
Jul 30 21:07:08.680866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:08.680837    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:07:08.681477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:08.681461    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:07:14.758603 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:14.758540    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:14.971866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:14.971833    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:19.581743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:19.581701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:25.048523 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:25.048484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:33.349073 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:33.348982    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:45.478231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:45.478197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:07:48.990955 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:07:48.990917    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:08:54.342863 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.342826    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b445799db-4z7zm"]
Jul 30 21:08:54.353350 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.353320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.356156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.356128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b445799db-4z7zm"]
Jul 30 21:08:54.453357 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.453319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-tls-cert\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.453525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.453382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx585\" (UniqueName: \"kubernetes.io/projected/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-kube-api-access-lx585\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.553922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.553881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx585\" (UniqueName: \"kubernetes.io/projected/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-kube-api-access-lx585\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.554124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.553979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-tls-cert\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.556548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.556522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-tls-cert\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.562172 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.562141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx585\" (UniqueName: \"kubernetes.io/projected/a742d93b-79cd-4e90-8d9d-c343ac9fd72c-kube-api-access-lx585\") pod \"authorino-7b445799db-4z7zm\" (UID: \"a742d93b-79cd-4e90-8d9d-c343ac9fd72c\") " pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.664464 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.664383    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b445799db-4z7zm"
Jul 30 21:08:54.801015 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.800927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b445799db-4z7zm"]
Jul 30 21:08:54.803956 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:08:54.803926    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda742d93b_79cd_4e90_8d9d_c343ac9fd72c.slice/crio-a4044c2a6143330556d3c0592877f0d95767347517a4f2598c68e7ffc841206a WatchSource:0}: Error finding container a4044c2a6143330556d3c0592877f0d95767347517a4f2598c68e7ffc841206a: Status 404 returned error can't find the container with id a4044c2a6143330556d3c0592877f0d95767347517a4f2598c68e7ffc841206a
Jul 30 21:08:54.805675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:54.805655    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:08:55.778932 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:55.778864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b445799db-4z7zm" event={"ID":"a742d93b-79cd-4e90-8d9d-c343ac9fd72c","Type":"ContainerStarted","Data":"1b29ca9d9b4dcf6819c38fa9c64195f3aa5562e934037d2c407143c05b192c27"}
Jul 30 21:08:55.779462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:55.778954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b445799db-4z7zm" event={"ID":"a742d93b-79cd-4e90-8d9d-c343ac9fd72c","Type":"ContainerStarted","Data":"a4044c2a6143330556d3c0592877f0d95767347517a4f2598c68e7ffc841206a"}
Jul 30 21:08:55.823428 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:55.823334    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b445799db-4z7zm" podStartSLOduration=1.208293497 podStartE2EDuration="1.823308636s" totalImagesPullingTime="615.015139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:08:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:08:55.795731095 +0000 UTC m=+707.561692975" watchObservedRunningTime="2026-07-30 21:08:55.823308636 +0000 UTC m=+707.589270516"
Jul 30 21:08:55.824985 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:55.824680    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:08:55.824985 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:55.824893    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cb899984c-q2hts" podUID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" containerName="authorino" containerID="cri-o://a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725" gracePeriod=30
Jul 30 21:08:56.083919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.083891    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:08:56.168857 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.168821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert\") pod \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") "
Jul 30 21:08:56.169058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.168889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdxw5\" (UniqueName: \"kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5\") pod \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\" (UID: \"6ee43205-c62a-4bdb-b85e-86bc653e2c82\") "
Jul 30 21:08:56.171592 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.171523    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5" (OuterVolumeSpecName: "kube-api-access-wdxw5") pod "6ee43205-c62a-4bdb-b85e-86bc653e2c82" (UID: "6ee43205-c62a-4bdb-b85e-86bc653e2c82"). InnerVolumeSpecName "kube-api-access-wdxw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:08:56.180319 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.180284    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6ee43205-c62a-4bdb-b85e-86bc653e2c82" (UID: "6ee43205-c62a-4bdb-b85e-86bc653e2c82"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:08:56.269947 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.269914    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ee43205-c62a-4bdb-b85e-86bc653e2c82-tls-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:08:56.269947 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.269942    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxw5\" (UniqueName: \"kubernetes.io/projected/6ee43205-c62a-4bdb-b85e-86bc653e2c82-kube-api-access-wdxw5\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:08:56.415532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.415294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:08:56.517089 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.517034    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:08:56.784315 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.784271    2583 generic.go:356] "Generic (PLEG): container finished" podID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" containerID="a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725" exitCode=0
Jul 30 21:08:56.784783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.784346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cb899984c-q2hts" event={"ID":"6ee43205-c62a-4bdb-b85e-86bc653e2c82","Type":"ContainerDied","Data":"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"}
Jul 30 21:08:56.784783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.784495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cb899984c-q2hts" event={"ID":"6ee43205-c62a-4bdb-b85e-86bc653e2c82","Type":"ContainerDied","Data":"eed46e6ee94c526e5f54f1c30f0d6c177a3406c433e3bc6d06e243aa78e1c104"}
Jul 30 21:08:56.784783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.784355    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cb899984c-q2hts"
Jul 30 21:08:56.784783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.784522    2583 scope.go:117] "RemoveContainer" containerID="a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"
Jul 30 21:08:56.795017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.794992    2583 scope.go:117] "RemoveContainer" containerID="a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"
Jul 30 21:08:56.795337 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:08:56.795316    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725\": container with ID starting with a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725 not found: ID does not exist" containerID="a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"
Jul 30 21:08:56.795421 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.795345    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725"} err="failed to get container status \"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725\": rpc error: code = NotFound desc = could not find container \"a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725\": container with ID starting with a1849b57dbbb3e9b1d371a1d685480a6521edaf7dcb256b57da6c62bfae0b725 not found: ID does not exist"
Jul 30 21:08:56.801452 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.801423    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:08:56.808465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:56.808439    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cb899984c-q2hts"]
Jul 30 21:08:57.445255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:57.445217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:08:57.546860 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:57.546830    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:08:58.777516 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:08:58.777481    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" path="/var/lib/kubelet/pods/6ee43205-c62a-4bdb-b85e-86bc653e2c82/volumes"
Jul 30 21:09:11.510682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:11.510645    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:09:43.034800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.034747    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:09:43.035413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.035010    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" podUID="d1f81515-b87b-4fb1-b85a-160c8505a61f" containerName="manager" containerID="cri-o://2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7" gracePeriod=30
Jul 30 21:09:43.294224 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.294173    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:09:43.360064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.360031    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk545\" (UniqueName: \"kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545\") pod \"d1f81515-b87b-4fb1-b85a-160c8505a61f\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") "
Jul 30 21:09:43.360221 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.360088    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert\") pod \"d1f81515-b87b-4fb1-b85a-160c8505a61f\" (UID: \"d1f81515-b87b-4fb1-b85a-160c8505a61f\") "
Jul 30 21:09:43.362329 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.362293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545" (OuterVolumeSpecName: "kube-api-access-dk545") pod "d1f81515-b87b-4fb1-b85a-160c8505a61f" (UID: "d1f81515-b87b-4fb1-b85a-160c8505a61f"). InnerVolumeSpecName "kube-api-access-dk545". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:09:43.362447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.362291    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d1f81515-b87b-4fb1-b85a-160c8505a61f" (UID: "d1f81515-b87b-4fb1-b85a-160c8505a61f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:09:43.461179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.461139    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk545\" (UniqueName: \"kubernetes.io/projected/d1f81515-b87b-4fb1-b85a-160c8505a61f-kube-api-access-dk545\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:09:43.461179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.461170    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d1f81515-b87b-4fb1-b85a-160c8505a61f-webhook-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:09:43.996662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.996622    2583 generic.go:356] "Generic (PLEG): container finished" podID="d1f81515-b87b-4fb1-b85a-160c8505a61f" containerID="2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7" exitCode=0
Jul 30 21:09:43.996834 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.996672    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7"
Jul 30 21:09:43.996834 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.996675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" event={"ID":"d1f81515-b87b-4fb1-b85a-160c8505a61f","Type":"ContainerDied","Data":"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"}
Jul 30 21:09:43.996834 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.996730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-rbtp7" event={"ID":"d1f81515-b87b-4fb1-b85a-160c8505a61f","Type":"ContainerDied","Data":"17120efe2f58280a0b657f146ea4cfba6dc6fb60a85c486b00eaef578e0bf7ef"}
Jul 30 21:09:43.996834 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:43.996762    2583 scope.go:117] "RemoveContainer" containerID="2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"
Jul 30 21:09:44.006268 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:44.006247    2583 scope.go:117] "RemoveContainer" containerID="2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"
Jul 30 21:09:44.006520 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:09:44.006503    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7\": container with ID starting with 2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7 not found: ID does not exist" containerID="2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"
Jul 30 21:09:44.006593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:44.006528    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7"} err="failed to get container status \"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7\": rpc error: code = NotFound desc = could not find container \"2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7\": container with ID starting with 2b5c33e0477c3f9a6e7c5fd929b8826ce23b98da57a827ac08b0392af8f13fc7 not found: ID does not exist"
Jul 30 21:09:44.015860 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:44.015832    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:09:44.021461 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:44.021435    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-rbtp7"]
Jul 30 21:09:44.776474 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:09:44.776438    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f81515-b87b-4fb1-b85a-160c8505a61f" path="/var/lib/kubelet/pods/d1f81515-b87b-4fb1-b85a-160c8505a61f/volumes"
Jul 30 21:10:44.768808 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.768757    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769118    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" containerName="authorino"
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769130    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" containerName="authorino"
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769148    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f81515-b87b-4fb1-b85a-160c8505a61f" containerName="manager"
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769154    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f81515-b87b-4fb1-b85a-160c8505a61f" containerName="manager"
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769209    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f81515-b87b-4fb1-b85a-160c8505a61f" containerName="manager"
Jul 30 21:10:44.769278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.769219    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ee43205-c62a-4bdb-b85e-86bc653e2c82" containerName="authorino"
Jul 30 21:10:44.772268 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.772247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.774281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.774257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 21:10:44.774677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.774654    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 21:10:44.774792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.774700    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67mks\""
Jul 30 21:10:44.786166 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.786139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:10:44.879006 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.878960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.879217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.879111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577jj\" (UniqueName: \"kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.980403 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.980370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577jj\" (UniqueName: \"kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.980537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.980429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.982901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.982875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:44.987977 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:44.987956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-577jj\" (UniqueName: \"kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj\") pod \"maas-controller-7d7c48cdc9-7ssrl\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") " pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:45.088065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:45.087985    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:45.222117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:45.222091    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:10:45.224130 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:10:45.224099    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fee3f45_2cde_47b1_8b14_119aa314d2fd.slice/crio-5fffe9cbdff56954ec816a4352333ead6adf59942f70ce8494a508bb20bd932b WatchSource:0}: Error finding container 5fffe9cbdff56954ec816a4352333ead6adf59942f70ce8494a508bb20bd932b: Status 404 returned error can't find the container with id 5fffe9cbdff56954ec816a4352333ead6adf59942f70ce8494a508bb20bd932b
Jul 30 21:10:45.261773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:45.261742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" event={"ID":"3fee3f45-2cde-47b1-8b14-119aa314d2fd","Type":"ContainerStarted","Data":"5fffe9cbdff56954ec816a4352333ead6adf59942f70ce8494a508bb20bd932b"}
Jul 30 21:10:46.267214 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:46.267172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" event={"ID":"3fee3f45-2cde-47b1-8b14-119aa314d2fd","Type":"ContainerStarted","Data":"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"}
Jul 30 21:10:46.267644 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:46.267303    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:10:46.283184 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:46.283127    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" podStartSLOduration=1.6420754 podStartE2EDuration="2.283112277s" totalImagesPullingTime="641.036877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:10:46.282277157 +0000 UTC m=+818.048239038" watchObservedRunningTime="2026-07-30 21:10:46.283112277 +0000 UTC m=+818.049074156"
Jul 30 21:10:57.273439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:10:57.273409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:11:01.802028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:01.801992    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:11:01.875556 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:01.875524    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:11:01.875756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:01.875733    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" podUID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" containerName="manager" containerID="cri-o://0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c" gracePeriod=30
Jul 30 21:11:02.122504 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.122479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:11:02.226726 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.226689    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert\") pod \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") "
Jul 30 21:11:02.226867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.226763    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-577jj\" (UniqueName: \"kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj\") pod \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\" (UID: \"3fee3f45-2cde-47b1-8b14-119aa314d2fd\") "
Jul 30 21:11:02.228893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.228854    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3fee3f45-2cde-47b1-8b14-119aa314d2fd" (UID: "3fee3f45-2cde-47b1-8b14-119aa314d2fd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:11:02.229003 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.228909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj" (OuterVolumeSpecName: "kube-api-access-577jj") pod "3fee3f45-2cde-47b1-8b14-119aa314d2fd" (UID: "3fee3f45-2cde-47b1-8b14-119aa314d2fd"). InnerVolumeSpecName "kube-api-access-577jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:11:02.327640 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.327603    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577jj\" (UniqueName: \"kubernetes.io/projected/3fee3f45-2cde-47b1-8b14-119aa314d2fd-kube-api-access-577jj\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:11:02.327640 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.327634    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fee3f45-2cde-47b1-8b14-119aa314d2fd-webhook-cert\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:11:02.340227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.340189    2583 generic.go:356] "Generic (PLEG): container finished" podID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" containerID="0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c" exitCode=0
Jul 30 21:11:02.340413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.340236    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl"
Jul 30 21:11:02.340413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.340269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" event={"ID":"3fee3f45-2cde-47b1-8b14-119aa314d2fd","Type":"ContainerDied","Data":"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"}
Jul 30 21:11:02.340413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.340340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-7ssrl" event={"ID":"3fee3f45-2cde-47b1-8b14-119aa314d2fd","Type":"ContainerDied","Data":"5fffe9cbdff56954ec816a4352333ead6adf59942f70ce8494a508bb20bd932b"}
Jul 30 21:11:02.340413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.340372    2583 scope.go:117] "RemoveContainer" containerID="0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"
Jul 30 21:11:02.349157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.349140    2583 scope.go:117] "RemoveContainer" containerID="0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"
Jul 30 21:11:02.349425 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:11:02.349405    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c\": container with ID starting with 0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c not found: ID does not exist" containerID="0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"
Jul 30 21:11:02.349476 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.349435    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c"} err="failed to get container status \"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c\": rpc error: code = NotFound desc = could not find container \"0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c\": container with ID starting with 0d4649e441face2903d2f85a52ef6b5c53c0584f79d1470959e26e97dc935e6c not found: ID does not exist"
Jul 30 21:11:02.359670 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.359619    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:11:02.364529 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.364505    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-7ssrl"]
Jul 30 21:11:02.776398 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:02.776366    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" path="/var/lib/kubelet/pods/3fee3f45-2cde-47b1-8b14-119aa314d2fd/volumes"
Jul 30 21:11:03.010323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.010292    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:11:03.110248 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.110165    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-zqkmv"]
Jul 30 21:11:03.110542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.110529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" containerName="manager"
Jul 30 21:11:03.110627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.110544    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" containerName="manager"
Jul 30 21:11:03.110627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.110611    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fee3f45-2cde-47b1-8b14-119aa314d2fd" containerName="manager"
Jul 30 21:11:03.113340 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.113325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.115655 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.115632    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 30 21:11:03.115780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.115633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-67mks\""
Jul 30 21:11:03.115780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.115633    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 30 21:11:03.122756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.122729    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-zqkmv"]
Jul 30 21:11:03.233944 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.233897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-webhook-cert\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.233944 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.233948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctfq\" (UniqueName: \"kubernetes.io/projected/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-kube-api-access-dctfq\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.334795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.334757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-webhook-cert\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.334968 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.334801    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctfq\" (UniqueName: \"kubernetes.io/projected/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-kube-api-access-dctfq\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.337343 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.337312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-webhook-cert\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.343895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.343869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctfq\" (UniqueName: \"kubernetes.io/projected/492f2e3f-3e4a-401f-aa2e-c40ee18b05f9-kube-api-access-dctfq\") pod \"maas-controller-7d7c48cdc9-zqkmv\" (UID: \"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9\") " pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.424361 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.424269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:03.558414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:03.558391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7c48cdc9-zqkmv"]
Jul 30 21:11:03.561137 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:11:03.561099    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f2e3f_3e4a_401f_aa2e_c40ee18b05f9.slice/crio-10756c1dd7dcf08323f7ee9ceb2885cc3289d78c92121bc85a6d9bf0c4a88571 WatchSource:0}: Error finding container 10756c1dd7dcf08323f7ee9ceb2885cc3289d78c92121bc85a6d9bf0c4a88571: Status 404 returned error can't find the container with id 10756c1dd7dcf08323f7ee9ceb2885cc3289d78c92121bc85a6d9bf0c4a88571
Jul 30 21:11:04.353974 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:04.353937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv" event={"ID":"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9","Type":"ContainerStarted","Data":"acc79b57225a591073d15d15dd5972b9b9668d5ab631587334d59cd046d55cbe"}
Jul 30 21:11:04.354347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:04.353990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv" event={"ID":"492f2e3f-3e4a-401f-aa2e-c40ee18b05f9","Type":"ContainerStarted","Data":"10756c1dd7dcf08323f7ee9ceb2885cc3289d78c92121bc85a6d9bf0c4a88571"}
Jul 30 21:11:04.354347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:04.354015    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:11:04.372998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:04.372949    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv" podStartSLOduration=0.778900066 podStartE2EDuration="1.372932965s" totalImagesPullingTime="594.032899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:11:04.370970076 +0000 UTC m=+836.136931955" watchObservedRunningTime="2026-07-30 21:11:04.372932965 +0000 UTC m=+836.138894844"
Jul 30 21:11:15.360343 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:11:15.360301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d7c48cdc9-zqkmv"
Jul 30 21:12:07.645138 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:07.645053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:07.748544 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:07.748501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:07.868926 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:07.868893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:08.707702 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:08.707674    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:12:08.708926 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:08.708903    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:12:09.151318 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:09.151278    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:09.216968 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:09.216932    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:13.533293 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:13.533256    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:13.597241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:13.597211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:13.676160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:13.676121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:13.749151 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:13.749110    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:13.876165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:13.876064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:19.534139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:19.534098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:12:19.634104 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:12:19.634059    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:13:05.200516 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:13:05.200480    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:15:00.139553 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.139469    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"]
Jul 30 21:15:00.142749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.142732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"
Jul 30 21:15:00.144596 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.144561    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:15:00.151739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.151722    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"
Jul 30 21:15:00.154894 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.154870    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"]
Jul 30 21:15:00.287307 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.287283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"]
Jul 30 21:15:00.289946 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:15:00.289912    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e86f508_6a66_4207_a4e0_0fb7e0283c60.slice/crio-9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e WatchSource:0}: Error finding container 9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e: Status 404 returned error can't find the container with id 9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e
Jul 30 21:15:00.291947 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.291927    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:15:00.416349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:00.416240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2" event={"ID":"0e86f508-6a66-4207-a4e0-0fb7e0283c60","Type":"ContainerStarted","Data":"9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e"}
Jul 30 21:15:03.430730 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:03.430690    2583 generic.go:356] "Generic (PLEG): container finished" podID="0e86f508-6a66-4207-a4e0-0fb7e0283c60" containerID="21cba2dbe2110b77c4018a2c210678a11481b511fd7ca1f369e26644a986dc18" exitCode=0
Jul 30 21:15:03.431110 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:03.430746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2" event={"ID":"0e86f508-6a66-4207-a4e0-0fb7e0283c60","Type":"ContainerDied","Data":"21cba2dbe2110b77c4018a2c210678a11481b511fd7ca1f369e26644a986dc18"}
Jul 30 21:15:04.573277 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:04.573246    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"
Jul 30 21:15:05.442885 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:05.442853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2"
Jul 30 21:15:05.442885 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:05.442877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757435-m28n2" event={"ID":"0e86f508-6a66-4207-a4e0-0fb7e0283c60","Type":"ContainerDied","Data":"9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e"}
Jul 30 21:15:05.443116 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:15:05.442933    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9060a6dd37799260426161394ca0313af25cf908a4c4b0f817448c5e77ad858e"
Jul 30 21:17:08.736782 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:08.736750    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:17:08.738699 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:08.738677    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:17:30.423525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.423491    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:17:30.426266 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.423732    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" podUID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" containerName="manager" containerID="cri-o://1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10" gracePeriod=10
Jul 30 21:17:30.675804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.675743    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:17:30.779749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.779716    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsr44\" (UniqueName: \"kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44\") pod \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") "
Jul 30 21:17:30.779899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.779769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume\") pod \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\" (UID: \"fa326a8e-f0d3-4f9c-a095-325fa526f0dc\") "
Jul 30 21:17:30.780211 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.780178    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fa326a8e-f0d3-4f9c-a095-325fa526f0dc" (UID: "fa326a8e-f0d3-4f9c-a095-325fa526f0dc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:17:30.781845 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.781818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44" (OuterVolumeSpecName: "kube-api-access-dsr44") pod "fa326a8e-f0d3-4f9c-a095-325fa526f0dc" (UID: "fa326a8e-f0d3-4f9c-a095-325fa526f0dc"). InnerVolumeSpecName "kube-api-access-dsr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:17:30.881121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.881085    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsr44\" (UniqueName: \"kubernetes.io/projected/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-kube-api-access-dsr44\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:17:30.881121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:30.881116    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa326a8e-f0d3-4f9c-a095-325fa526f0dc-extensions-socket-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:17:31.121005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.120971    2583 generic.go:356] "Generic (PLEG): container finished" podID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" containerID="1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10" exitCode=0
Jul 30 21:17:31.121005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.121012    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" event={"ID":"fa326a8e-f0d3-4f9c-a095-325fa526f0dc","Type":"ContainerDied","Data":"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"}
Jul 30 21:17:31.121245 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.121017    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"
Jul 30 21:17:31.121245 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.121051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf" event={"ID":"fa326a8e-f0d3-4f9c-a095-325fa526f0dc","Type":"ContainerDied","Data":"8baad7cbb95460a8be72c9ce57ed2055acb34d4007d96abe9df317243041b10d"}
Jul 30 21:17:31.121245 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.121073    2583 scope.go:117] "RemoveContainer" containerID="1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"
Jul 30 21:17:31.131134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.131115    2583 scope.go:117] "RemoveContainer" containerID="1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"
Jul 30 21:17:31.131392 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:17:31.131375    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10\": container with ID starting with 1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10 not found: ID does not exist" containerID="1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"
Jul 30 21:17:31.131484 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.131400    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10"} err="failed to get container status \"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10\": rpc error: code = NotFound desc = could not find container \"1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10\": container with ID starting with 1a641b7c2de8376249b69ea8a004ed0d664d80ffaa3afb7686dbcdab30891e10 not found: ID does not exist"
Jul 30 21:17:31.141815 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.141787    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:17:31.145541 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:31.145519    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-sqcxf"]
Jul 30 21:17:32.776748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:17:32.776711    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" path="/var/lib/kubelet/pods/fa326a8e-f0d3-4f9c-a095-325fa526f0dc/volumes"
Jul 30 21:18:31.126410 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126374    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:18:31.126914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e86f508-6a66-4207-a4e0-0fb7e0283c60" containerName="cleanup"
Jul 30 21:18:31.126914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126807    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e86f508-6a66-4207-a4e0-0fb7e0283c60" containerName="cleanup"
Jul 30 21:18:31.126914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126828    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" containerName="manager"
Jul 30 21:18:31.126914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126833    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" containerName="manager"
Jul 30 21:18:31.127066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126932    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e86f508-6a66-4207-a4e0-0fb7e0283c60" containerName="cleanup"
Jul 30 21:18:31.127066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.126942    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa326a8e-f0d3-4f9c-a095-325fa526f0dc" containerName="manager"
Jul 30 21:18:31.130848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.130829    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.132832 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.132814    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6h2kb\""
Jul 30 21:18:31.141508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.141489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:18:31.257725 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.257694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsc8l\" (UniqueName: \"kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.257889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.257744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.358770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.358718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsc8l\" (UniqueName: \"kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.358990 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.358795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.359177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.359156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.367473 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.367442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsc8l\" (UniqueName: \"kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l\") pod \"kuadrant-operator-controller-manager-68cf589589-htxtn\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.441493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.441396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:31.580327 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:31.580280    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:18:31.581600 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:18:31.581555    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe5e9ec_bf33_4abe_b7bc_4bf573b1fada.slice/crio-27c3df7d3dfbf36931d5ef8166a3afaeeb19cb49e315a4f9ab00082f351c1208 WatchSource:0}: Error finding container 27c3df7d3dfbf36931d5ef8166a3afaeeb19cb49e315a4f9ab00082f351c1208: Status 404 returned error can't find the container with id 27c3df7d3dfbf36931d5ef8166a3afaeeb19cb49e315a4f9ab00082f351c1208
Jul 30 21:18:32.394312 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:32.394270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" event={"ID":"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada","Type":"ContainerStarted","Data":"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"}
Jul 30 21:18:32.394773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:32.394366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:18:32.394773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:32.394385    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" event={"ID":"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada","Type":"ContainerStarted","Data":"27c3df7d3dfbf36931d5ef8166a3afaeeb19cb49e315a4f9ab00082f351c1208"}
Jul 30 21:18:32.417002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:32.416939    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" podStartSLOduration=1.416920401 podStartE2EDuration="1.416920401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:18:32.415276552 +0000 UTC m=+1284.181238432" watchObservedRunningTime="2026-07-30 21:18:32.416920401 +0000 UTC m=+1284.182882280"
Jul 30 21:18:43.401283 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:18:43.401253    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:19:03.964524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:19:03.964485    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:19:06.744501 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:19:06.744462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:20:38.780469 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:20:38.780439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:20:58.091170 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:20:58.091132    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:20:58.192588 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:20:58.192528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:21:40.940636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.940533    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:21:40.943943 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.943919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.945771 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.945751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-72e4ee54-kserve-self-signed-certs\""
Jul 30 21:21:40.950897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.950874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.951022 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.950915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.951022 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.950973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.951022 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.950999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.951158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.951090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2wq\" (UniqueName: \"kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.951158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.951128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:40.962114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.962082    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:21:40.978133 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:40.978097    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:21:41.052310 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052310 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2wq\" (UniqueName: \"kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052867    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.052958 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.052936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.054723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.054701    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.055033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.055017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.060304 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.060278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2wq\" (UniqueName: \"kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq\") pod \"e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") " pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.255674 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.255638    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:41.393379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:41.393314    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:21:41.396134 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:21:41.396103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55fb103e_c56e_4902_bf4e_ef8fe552e6eb.slice/crio-c8d509d87cabd1c49d5e3f12846f153429d03bc3fd32a890983db1e04c9e4e0f WatchSource:0}: Error finding container c8d509d87cabd1c49d5e3f12846f153429d03bc3fd32a890983db1e04c9e4e0f: Status 404 returned error can't find the container with id c8d509d87cabd1c49d5e3f12846f153429d03bc3fd32a890983db1e04c9e4e0f
Jul 30 21:21:42.267109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:42.267070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" event={"ID":"55fb103e-c56e-4902-bf4e-ef8fe552e6eb","Type":"ContainerStarted","Data":"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"}
Jul 30 21:21:42.267500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:42.267120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" event={"ID":"55fb103e-c56e-4902-bf4e-ef8fe552e6eb","Type":"ContainerStarted","Data":"c8d509d87cabd1c49d5e3f12846f153429d03bc3fd32a890983db1e04c9e4e0f"}
Jul 30 21:21:42.285574 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:42.285504    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" podStartSLOduration=2.285483419 podStartE2EDuration="2.285483419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:21:42.283389352 +0000 UTC m=+1474.049351232" watchObservedRunningTime="2026-07-30 21:21:42.285483419 +0000 UTC m=+1474.051445302"
Jul 30 21:21:48.947453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:48.947421    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:21:48.951349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:48.951326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:48.953229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:48.953207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-72e4ee54-kserve-self-signed-certs\""
Jul 30 21:21:48.963761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:48.963737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:21:48.987077 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:48.987028    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:21:49.022048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.022217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022075    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.022217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.022217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.022349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hvsb\" (UniqueName: \"kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.022349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.022262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.122987 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.122953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hvsb\" (UniqueName: \"kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.123714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.123641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.125493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.125462    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.125744 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.125727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.131190 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.131158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hvsb\" (UniqueName: \"kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb\") pod \"e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") " pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.261969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.261931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:49.398742 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:49.398678    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:21:49.401115 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:21:49.401087    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9656a6c0_b725_4e73_8317_5720306ef753.slice/crio-4133bce44c7403006fab7cb9f2d43372f09468afebd2adeb3beb77924c60c9f9 WatchSource:0}: Error finding container 4133bce44c7403006fab7cb9f2d43372f09468afebd2adeb3beb77924c60c9f9: Status 404 returned error can't find the container with id 4133bce44c7403006fab7cb9f2d43372f09468afebd2adeb3beb77924c60c9f9
Jul 30 21:21:50.306871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:50.306834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" event={"ID":"9656a6c0-b725-4e73-8317-5720306ef753","Type":"ContainerStarted","Data":"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"}
Jul 30 21:21:50.306871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:50.306877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" event={"ID":"9656a6c0-b725-4e73-8317-5720306ef753","Type":"ContainerStarted","Data":"4133bce44c7403006fab7cb9f2d43372f09468afebd2adeb3beb77924c60c9f9"}
Jul 30 21:21:50.323201 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:50.323151    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" podStartSLOduration=2.323136807 podStartE2EDuration="2.323136807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:21:50.322005639 +0000 UTC m=+1482.087967529" watchObservedRunningTime="2026-07-30 21:21:50.323136807 +0000 UTC m=+1482.089098685"
Jul 30 21:21:51.256174 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:51.256135    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:51.256415 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:51.256197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:51.269514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:51.269484    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:51.325160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:51.325125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:21:59.262660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:59.262627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:59.262660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:59.262671    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:59.275104 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:59.275068    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:21:59.357041 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:21:59.357011    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:22:08.767876 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:08.767848    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:22:08.769673 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:08.769648    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:22:17.931726 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:17.931686    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:22:17.955032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:17.955000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:22:19.499341 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:19.499306    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:22:19.518586 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:19.518528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:22:32.951244 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:32.951181    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" podUID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" containerName="main" containerID="cri-o://cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157" gracePeriod=45
Jul 30 21:22:33.203774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.203719    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:22:33.290435 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.290400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.290660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.290467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.290792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.290760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:33.292878 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.292856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm" (OuterVolumeSpecName: "dshm") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:33.391008 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.390974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.391008 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391009    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd2wq\" (UniqueName: \"kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.391256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391031    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.391256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391053    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home\") pod \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\" (UID: \"55fb103e-c56e-4902-bf4e-ef8fe552e6eb\") "
Jul 30 21:22:33.391256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391209    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.391256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391224    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.391451 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391304    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:33.391451 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.391362    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home" (OuterVolumeSpecName: "home") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:33.393354 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.393323    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq" (OuterVolumeSpecName: "kube-api-access-xd2wq") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "kube-api-access-xd2wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:22:33.393354 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.393340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55fb103e-c56e-4902-bf4e-ef8fe552e6eb" (UID: "55fb103e-c56e-4902-bf4e-ef8fe552e6eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:22:33.491952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.491915    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.491952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.491947    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.491952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.491958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2wq\" (UniqueName: \"kubernetes.io/projected/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-kube-api-access-xd2wq\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.492175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.491967    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55fb103e-c56e-4902-bf4e-ef8fe552e6eb-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:33.507158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.507122    2583 generic.go:356] "Generic (PLEG): container finished" podID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" containerID="cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157" exitCode=0
Jul 30 21:22:33.507289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.507167    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"
Jul 30 21:22:33.507289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.507209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" event={"ID":"55fb103e-c56e-4902-bf4e-ef8fe552e6eb","Type":"ContainerDied","Data":"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"}
Jul 30 21:22:33.507289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.507260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w" event={"ID":"55fb103e-c56e-4902-bf4e-ef8fe552e6eb","Type":"ContainerDied","Data":"c8d509d87cabd1c49d5e3f12846f153429d03bc3fd32a890983db1e04c9e4e0f"}
Jul 30 21:22:33.507289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.507279    2583 scope.go:117] "RemoveContainer" containerID="cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"
Jul 30 21:22:33.515930 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.515914    2583 scope.go:117] "RemoveContainer" containerID="cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"
Jul 30 21:22:33.516178 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:22:33.516160    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157\": container with ID starting with cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157 not found: ID does not exist" containerID="cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"
Jul 30 21:22:33.516223 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.516186    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157"} err="failed to get container status \"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157\": rpc error: code = NotFound desc = could not find container \"cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157\": container with ID starting with cfbf8ed58f8f4ba9b402402205aeba3e7bd152a1f787c64acdbcac3f9fa6f157 not found: ID does not exist"
Jul 30 21:22:33.527978 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.527958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:22:33.539306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:33.539279    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-72e4ee54-kserve-5944cc7679-8tf6w"]
Jul 30 21:22:34.519291 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.519228    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" podUID="9656a6c0-b725-4e73-8317-5720306ef753" containerName="main" containerID="cri-o://4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2" gracePeriod=45
Jul 30 21:22:34.776910 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.776847    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" path="/var/lib/kubelet/pods/55fb103e-c56e-4902-bf4e-ef8fe552e6eb/volumes"
Jul 30 21:22:34.777529 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.777504    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:22:34.801531 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801501    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.801716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.801716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801598    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.801716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801619    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hvsb\" (UniqueName: \"kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.801716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801658    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.801716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs\") pod \"9656a6c0-b725-4e73-8317-5720306ef753\" (UID: \"9656a6c0-b725-4e73-8317-5720306ef753\") "
Jul 30 21:22:34.802050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.801998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:34.802050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.802004    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home" (OuterVolumeSpecName: "home") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:34.802367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.802327    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache" (OuterVolumeSpecName: "model-cache") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:34.803945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.803919    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm" (OuterVolumeSpecName: "dshm") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:22:34.804226 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.804193    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:22:34.804496 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.804471    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb" (OuterVolumeSpecName: "kube-api-access-7hvsb") pod "9656a6c0-b725-4e73-8317-5720306ef753" (UID: "9656a6c0-b725-4e73-8317-5720306ef753"). InnerVolumeSpecName "kube-api-access-7hvsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:22:34.902642 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902605    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:34.902642 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902638    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:34.902642 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902649    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hvsb\" (UniqueName: \"kubernetes.io/projected/9656a6c0-b725-4e73-8317-5720306ef753-kube-api-access-7hvsb\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:34.902887 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902660    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:34.902887 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902669    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9656a6c0-b725-4e73-8317-5720306ef753-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:34.902887 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:34.902676    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9656a6c0-b725-4e73-8317-5720306ef753-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:22:35.521808 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.521768    2583 generic.go:356] "Generic (PLEG): container finished" podID="9656a6c0-b725-4e73-8317-5720306ef753" containerID="4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2" exitCode=0
Jul 30 21:22:35.522227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.521807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" event={"ID":"9656a6c0-b725-4e73-8317-5720306ef753","Type":"ContainerDied","Data":"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"}
Jul 30 21:22:35.522227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.521831    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"
Jul 30 21:22:35.522227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.521853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8" event={"ID":"9656a6c0-b725-4e73-8317-5720306ef753","Type":"ContainerDied","Data":"4133bce44c7403006fab7cb9f2d43372f09468afebd2adeb3beb77924c60c9f9"}
Jul 30 21:22:35.522227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.521872    2583 scope.go:117] "RemoveContainer" containerID="4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"
Jul 30 21:22:35.531453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.531432    2583 scope.go:117] "RemoveContainer" containerID="4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"
Jul 30 21:22:35.531743 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:22:35.531721    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2\": container with ID starting with 4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2 not found: ID does not exist" containerID="4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"
Jul 30 21:22:35.531840 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.531748    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2"} err="failed to get container status \"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2\": rpc error: code = NotFound desc = could not find container \"4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2\": container with ID starting with 4ad2741b2ddc6ba6ee2202b16791f895165e3b164ab92e833c54fd1a50580bf2 not found: ID does not exist"
Jul 30 21:22:35.542859 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.542834    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:22:35.544913 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:35.544892    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-72e4ee54-kserve-5544c4dd59-tzxk8"]
Jul 30 21:22:36.777188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:22:36.777153    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9656a6c0-b725-4e73-8317-5720306ef753" path="/var/lib/kubelet/pods/9656a6c0-b725-4e73-8317-5720306ef753/volumes"
Jul 30 21:23:29.444960 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:23:29.444906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:24:06.417053 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:24:06.416964    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:25:42.104091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:25:42.103973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:08.194024 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:08.193978    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:11.504400 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:11.504357    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:11.605861 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:11.605825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:11.960292 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:11.960198    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:17.534780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:17.534708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:17.665860 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:17.665823    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:17.768200 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:17.768163    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:18.129555 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:18.129513    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:18.228561 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:18.228521    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:20.245507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:20.245474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:20.343664 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:20.343618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:21.438764 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:21.438726    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:21.617408 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:21.617364    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:22.054484 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.054446    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:22.151093 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.151045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:26:22.167193 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167163    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:26:22.167497 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167486    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9656a6c0-b725-4e73-8317-5720306ef753" containerName="main"
Jul 30 21:26:22.167543 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167499    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9656a6c0-b725-4e73-8317-5720306ef753" containerName="main"
Jul 30 21:26:22.167543 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167508    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" containerName="main"
Jul 30 21:26:22.167543 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167514    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" containerName="main"
Jul 30 21:26:22.167674 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167582    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="55fb103e-c56e-4902-bf4e-ef8fe552e6eb" containerName="main"
Jul 30 21:26:22.167674 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.167592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9656a6c0-b725-4e73-8317-5720306ef753" containerName="main"
Jul 30 21:26:22.170598 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.170553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.172548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.172527    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1b7f48d9-dockercfg-sstgb\""
Jul 30 21:26:22.172548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.172542    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1b7f48d9\""
Jul 30 21:26:22.183300 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.183279    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:26:22.195059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.195034    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:26:22.198496 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.198473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.206037 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.206013    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:26:22.323058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.322963    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.323058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.323024    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmgw\" (UniqueName: \"kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.323255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.323111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.323255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.323197    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2msj\" (UniqueName: \"kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.423986 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.423940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.424188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.423994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmgw\" (UniqueName: \"kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.424188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.424028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.424188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.424086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2msj\" (UniqueName: \"kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.424754 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.424734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.424816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.424783    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.431175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.431149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmgw\" (UniqueName: \"kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw\") pod \"payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.431271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.431155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2msj\" (UniqueName: \"kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj\") pod \"payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") " pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.482145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.482104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:22.510491 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.510459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:22.641729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.641658    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:26:22.645386 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:26:22.645312    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341ea8b5_2ef0_450f_a528_5ca534574fb4.slice/crio-a2de94d85559ebb37399f31c1c012186e03ba34c7ebae26a1e5061364457e3cb WatchSource:0}: Error finding container a2de94d85559ebb37399f31c1c012186e03ba34c7ebae26a1e5061364457e3cb: Status 404 returned error can't find the container with id a2de94d85559ebb37399f31c1c012186e03ba34c7ebae26a1e5061364457e3cb
Jul 30 21:26:22.669419 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:22.669395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:26:22.679666 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:26:22.679637    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aa0501_d9c2_4cf5_82cc_6ae2a667b390.slice/crio-b7116ae4bf8334a19b38ddf75af466e0e737432a522e1eb30a93883d709e4773 WatchSource:0}: Error finding container b7116ae4bf8334a19b38ddf75af466e0e737432a522e1eb30a93883d709e4773: Status 404 returned error can't find the container with id b7116ae4bf8334a19b38ddf75af466e0e737432a522e1eb30a93883d709e4773
Jul 30 21:26:23.598715 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.598678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" event={"ID":"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390","Type":"ContainerStarted","Data":"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"}
Jul 30 21:26:23.599323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.598748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" event={"ID":"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390","Type":"ContainerStarted","Data":"b7116ae4bf8334a19b38ddf75af466e0e737432a522e1eb30a93883d709e4773"}
Jul 30 21:26:23.599323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.598809    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:23.600496 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.600474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" event={"ID":"341ea8b5-2ef0-450f-a528-5ca534574fb4","Type":"ContainerStarted","Data":"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"}
Jul 30 21:26:23.600632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.600515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" event={"ID":"341ea8b5-2ef0-450f-a528-5ca534574fb4","Type":"ContainerStarted","Data":"a2de94d85559ebb37399f31c1c012186e03ba34c7ebae26a1e5061364457e3cb"}
Jul 30 21:26:23.600632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.600584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:23.612390 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.612339    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" podStartSLOduration=1.6123125310000002 podStartE2EDuration="1.612312531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:26:23.611992099 +0000 UTC m=+1755.377953978" watchObservedRunningTime="2026-07-30 21:26:23.612312531 +0000 UTC m=+1755.378274411"
Jul 30 21:26:23.625223 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:23.625181    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" podStartSLOduration=1.625163133 podStartE2EDuration="1.625163133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:26:23.624469839 +0000 UTC m=+1755.390431931" watchObservedRunningTime="2026-07-30 21:26:23.625163133 +0000 UTC m=+1755.391125021"
Jul 30 21:26:34.608125 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:34.608093    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:26:34.608652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:34.608179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:26:51.852043 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:51.852009    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"]
Jul 30 21:26:51.855364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:51.855348    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:51.857265 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:51.857244    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:26:51.861835 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:51.861808    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"]
Jul 30 21:26:51.959621 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:51.959556    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69\" (UID: \"2f1cabcd-b13e-42a2-8d51-ddc832488c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:52.061082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.061043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69\" (UID: \"2f1cabcd-b13e-42a2-8d51-ddc832488c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:52.061746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.061726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69\" (UID: \"2f1cabcd-b13e-42a2-8d51-ddc832488c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:52.167528 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.167448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:52.303066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.303037    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"]
Jul 30 21:26:52.305314 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:26:52.305289    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1cabcd_b13e_42a2_8d51_ddc832488c04.slice/crio-bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374 WatchSource:0}: Error finding container bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374: Status 404 returned error can't find the container with id bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374
Jul 30 21:26:52.739863 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.739829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69" event={"ID":"2f1cabcd-b13e-42a2-8d51-ddc832488c04","Type":"ContainerStarted","Data":"97d49e9c90fc91bf6a3eca81a48790e8a30ddaea1a2d4bd99a0e32ffdbb84ba7"}
Jul 30 21:26:52.740047 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.739892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69" event={"ID":"2f1cabcd-b13e-42a2-8d51-ddc832488c04","Type":"ContainerStarted","Data":"bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374"}
Jul 30 21:26:52.754358 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:52.754256    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69" podStartSLOduration=1.754240579 podStartE2EDuration="1.754240579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:26:52.752709605 +0000 UTC m=+1784.518671485" watchObservedRunningTime="2026-07-30 21:26:52.754240579 +0000 UTC m=+1784.520202521"
Jul 30 21:26:53.745485 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:53.745448    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f1cabcd-b13e-42a2-8d51-ddc832488c04" containerID="97d49e9c90fc91bf6a3eca81a48790e8a30ddaea1a2d4bd99a0e32ffdbb84ba7" exitCode=0
Jul 30 21:26:53.745944 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:53.745548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69" event={"ID":"2f1cabcd-b13e-42a2-8d51-ddc832488c04","Type":"ContainerDied","Data":"97d49e9c90fc91bf6a3eca81a48790e8a30ddaea1a2d4bd99a0e32ffdbb84ba7"}
Jul 30 21:26:54.887686 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:54.887661    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:54.986707 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:54.986675    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca\") pod \"2f1cabcd-b13e-42a2-8d51-ddc832488c04\" (UID: \"2f1cabcd-b13e-42a2-8d51-ddc832488c04\") "
Jul 30 21:26:54.987020 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:54.986999    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f1cabcd-b13e-42a2-8d51-ddc832488c04" (UID: "2f1cabcd-b13e-42a2-8d51-ddc832488c04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:26:55.087892 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:55.087788    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cabcd-b13e-42a2-8d51-ddc832488c04-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:26:55.755790 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:55.755759    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"
Jul 30 21:26:55.755973 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:55.755789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69" event={"ID":"2f1cabcd-b13e-42a2-8d51-ddc832488c04","Type":"ContainerDied","Data":"bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374"}
Jul 30 21:26:55.755973 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:26:55.755843    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd336abba31835455ee6af36bc0d45a0f2c8020e3e0ec53e30dd26637e053374"
Jul 30 21:27:02.288877 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.288779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:02.312998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.312964    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:27:02.313200 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.313176    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" podUID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" containerName="payload-processing" containerID="cri-o://ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294" gracePeriod=30
Jul 30 21:27:02.320411 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.320257    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:27:02.320534 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.320489    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" podUID="341ea8b5-2ef0-450f-a528-5ca534574fb4" containerName="payload-pre-processing" containerID="cri-o://9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4" gracePeriod=30
Jul 30 21:27:02.388731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.388695    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:02.454606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.454543    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"]
Jul 30 21:27:02.456855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.456809    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:02.457005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.456891    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1b7f48d9-zqw69"]
Jul 30 21:27:02.511816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.511786    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:02.593382 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.593358    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:27:02.615436 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.615408    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:27:02.640033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.639979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:02.655863 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.655832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2msj\" (UniqueName: \"kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj\") pod \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") "
Jul 30 21:27:02.656035 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.655925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume\") pod \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\" (UID: \"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390\") "
Jul 30 21:27:02.656301 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.656281    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" (UID: "e3aa0501-d9c2-4cf5-82cc-6ae2a667b390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:02.658025 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.658003    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj" (OuterVolumeSpecName: "kube-api-access-p2msj") pod "e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" (UID: "e3aa0501-d9c2-4cf5-82cc-6ae2a667b390"). InnerVolumeSpecName "kube-api-access-p2msj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:27:02.757405 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.757363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume\") pod \"341ea8b5-2ef0-450f-a528-5ca534574fb4\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") "
Jul 30 21:27:02.757627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.757443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfmgw\" (UniqueName: \"kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw\") pod \"341ea8b5-2ef0-450f-a528-5ca534574fb4\" (UID: \"341ea8b5-2ef0-450f-a528-5ca534574fb4\") "
Jul 30 21:27:02.757682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.757646    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2msj\" (UniqueName: \"kubernetes.io/projected/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-kube-api-access-p2msj\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:02.757682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.757667    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:02.757772 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.757745    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "341ea8b5-2ef0-450f-a528-5ca534574fb4" (UID: "341ea8b5-2ef0-450f-a528-5ca534574fb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:02.759771 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.759742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw" (OuterVolumeSpecName: "kube-api-access-tfmgw") pod "341ea8b5-2ef0-450f-a528-5ca534574fb4" (UID: "341ea8b5-2ef0-450f-a528-5ca534574fb4"). InnerVolumeSpecName "kube-api-access-tfmgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:27:02.781579 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.781509    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1cabcd-b13e-42a2-8d51-ddc832488c04" path="/var/lib/kubelet/pods/2f1cabcd-b13e-42a2-8d51-ddc832488c04/volumes"
Jul 30 21:27:02.790220 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.790175    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" containerID="ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294" exitCode=0
Jul 30 21:27:02.790432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.790248    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"
Jul 30 21:27:02.790432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.790250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" event={"ID":"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390","Type":"ContainerDied","Data":"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"}
Jul 30 21:27:02.790432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.790310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh" event={"ID":"e3aa0501-d9c2-4cf5-82cc-6ae2a667b390","Type":"ContainerDied","Data":"b7116ae4bf8334a19b38ddf75af466e0e737432a522e1eb30a93883d709e4773"}
Jul 30 21:27:02.790432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.790358    2583 scope.go:117] "RemoveContainer" containerID="ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"
Jul 30 21:27:02.792299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.792249    2583 generic.go:356] "Generic (PLEG): container finished" podID="341ea8b5-2ef0-450f-a528-5ca534574fb4" containerID="9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4" exitCode=0
Jul 30 21:27:02.792462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.792368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" event={"ID":"341ea8b5-2ef0-450f-a528-5ca534574fb4","Type":"ContainerDied","Data":"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"}
Jul 30 21:27:02.792462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.792434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl" event={"ID":"341ea8b5-2ef0-450f-a528-5ca534574fb4","Type":"ContainerDied","Data":"a2de94d85559ebb37399f31c1c012186e03ba34c7ebae26a1e5061364457e3cb"}
Jul 30 21:27:02.792589 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.792491    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"
Jul 30 21:27:02.802194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.802174    2583 scope.go:117] "RemoveContainer" containerID="ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"
Jul 30 21:27:02.802546 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:27:02.802525    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294\": container with ID starting with ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294 not found: ID does not exist" containerID="ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"
Jul 30 21:27:02.802675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.802558    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294"} err="failed to get container status \"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294\": rpc error: code = NotFound desc = could not find container \"ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294\": container with ID starting with ceae28b6bb7663dcdf9e7dcb5516edc6614c84a095e3a59043df47913a4e7294 not found: ID does not exist"
Jul 30 21:27:02.802675 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.802602    2583 scope.go:117] "RemoveContainer" containerID="9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"
Jul 30 21:27:02.811881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.811644    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:27:02.812439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.812416    2583 scope.go:117] "RemoveContainer" containerID="9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"
Jul 30 21:27:02.812866 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:27:02.812842    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4\": container with ID starting with 9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4 not found: ID does not exist" containerID="9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"
Jul 30 21:27:02.812963 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.812873    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4"} err="failed to get container status \"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4\": rpc error: code = NotFound desc = could not find container \"9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4\": container with ID starting with 9db293dee2aa4b8f554d2a00912591e0711af4118da4c326e64014334aea68d4 not found: ID does not exist"
Jul 30 21:27:02.813698 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.813675    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1b7f48d9-7484cbf496-sg7pl"]
Jul 30 21:27:02.821889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.821848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:27:02.824077 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.824043    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1b7f48d9-5ffc587b74-nswgh"]
Jul 30 21:27:02.858317 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.858214    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfmgw\" (UniqueName: \"kubernetes.io/projected/341ea8b5-2ef0-450f-a528-5ca534574fb4-kube-api-access-tfmgw\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:02.858317 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.858252    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341ea8b5-2ef0-450f-a528-5ca534574fb4-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:02.876452 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:02.876403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:03.155963 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:03.155878    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:03.254334 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:03.254293    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:03.393685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:03.393634    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:03.746086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:03.746049    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:03.992062 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:03.992025    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:04.776684 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:04.776644    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341ea8b5-2ef0-450f-a528-5ca534574fb4" path="/var/lib/kubelet/pods/341ea8b5-2ef0-450f-a528-5ca534574fb4/volumes"
Jul 30 21:27:04.777186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:04.777139    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" path="/var/lib/kubelet/pods/e3aa0501-d9c2-4cf5-82cc-6ae2a667b390/volumes"
Jul 30 21:27:06.660240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.660204    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:06.732081 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.732044    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:06.733239 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733210    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" containerName="payload-processing"
Jul 30 21:27:06.733394 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733380    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" containerName="payload-processing"
Jul 30 21:27:06.733548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733533    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341ea8b5-2ef0-450f-a528-5ca534574fb4" containerName="payload-pre-processing"
Jul 30 21:27:06.733688 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733675    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="341ea8b5-2ef0-450f-a528-5ca534574fb4" containerName="payload-pre-processing"
Jul 30 21:27:06.733790 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733778    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1cabcd-b13e-42a2-8d51-ddc832488c04" containerName="revoke-keys"
Jul 30 21:27:06.733881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.733870    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1cabcd-b13e-42a2-8d51-ddc832488c04" containerName="revoke-keys"
Jul 30 21:27:06.734145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.734122    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1cabcd-b13e-42a2-8d51-ddc832488c04" containerName="revoke-keys"
Jul 30 21:27:06.734282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.734269    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="341ea8b5-2ef0-450f-a528-5ca534574fb4" containerName="payload-pre-processing"
Jul 30 21:27:06.734378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.734367    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3aa0501-d9c2-4cf5-82cc-6ae2a667b390" containerName="payload-processing"
Jul 30 21:27:06.740131 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.740102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.742075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.742049    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-60d76360-serving-cert\""
Jul 30 21:27:06.742170 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.742120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:27:06.745313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.745288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:06.767098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.767061    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:06.889325 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.889285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.889512 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.889346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrvk\" (UniqueName: \"kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.889512 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.889378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.935792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.935762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:06.990079 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.990052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.990242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.990092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrvk\" (UniqueName: \"kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.990242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.990113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.990731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.990660    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.992407 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.992387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:06.998459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:06.998434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrvk\" (UniqueName: \"kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk\") pod \"maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:07.054118 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.054083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:07.195554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.195472    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:07.199969 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:07.199911    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347c59fd_c9f3_45b7_a113_2e86d3c6e0c4.slice/crio-4a521858e5d2dedaf1d9466b591311cc745f7f6ce9d087edfb8db556fbfc12b7 WatchSource:0}: Error finding container 4a521858e5d2dedaf1d9466b591311cc745f7f6ce9d087edfb8db556fbfc12b7: Status 404 returned error can't find the container with id 4a521858e5d2dedaf1d9466b591311cc745f7f6ce9d087edfb8db556fbfc12b7
Jul 30 21:27:07.201639 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.201610    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:27:07.274843 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.274797    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:07.382678 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.382644    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:07.391106 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.391073    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:07.395462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.395442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.397542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.397335    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-60d76360-dockercfg-rx4dv\""
Jul 30 21:27:07.397761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.397746    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-60d76360\""
Jul 30 21:27:07.401707 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.401684    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:07.412124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.412101    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:07.415810 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.415788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.427203 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.427174    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:07.495322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.495281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.495619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.495461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn6v\" (UniqueName: \"kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.495619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.495544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.495619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.495612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrsw\" (UniqueName: \"kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.596423 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.596390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn6v\" (UniqueName: \"kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.596649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.596445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.596649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.596471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrsw\" (UniqueName: \"kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.596649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.596501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.597168 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.597145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.597259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.597236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.604070 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.604044    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn6v\" (UniqueName: \"kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v\") pod \"payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") " pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.604193 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.604103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrsw\" (UniqueName: \"kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw\") pod \"payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.707380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.707344    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:07.727518 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.727485    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:07.820472 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.820434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" event={"ID":"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4","Type":"ContainerStarted","Data":"4a521858e5d2dedaf1d9466b591311cc745f7f6ce9d087edfb8db556fbfc12b7"}
Jul 30 21:27:07.857192 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.857165    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:07.859012 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:07.858986    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512943c9_5477_4907_b60b_d77d89bffbba.slice/crio-89b2a5865a9dee1f64ebe298c2cbf3bce6bb59dadd53eb1238fc7c3e34bd35c9 WatchSource:0}: Error finding container 89b2a5865a9dee1f64ebe298c2cbf3bce6bb59dadd53eb1238fc7c3e34bd35c9: Status 404 returned error can't find the container with id 89b2a5865a9dee1f64ebe298c2cbf3bce6bb59dadd53eb1238fc7c3e34bd35c9
Jul 30 21:27:07.880466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:07.880443    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:07.888876 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:07.888803    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171a9755_6e90_4f00_b28b_6bf2c8a2ee68.slice/crio-407318a38fc05935cfd24f6fa65acd1a2c45f01e07af7937786f4735cea8caaf WatchSource:0}: Error finding container 407318a38fc05935cfd24f6fa65acd1a2c45f01e07af7937786f4735cea8caaf: Status 404 returned error can't find the container with id 407318a38fc05935cfd24f6fa65acd1a2c45f01e07af7937786f4735cea8caaf
Jul 30 21:27:08.828904 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.828863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" event={"ID":"512943c9-5477-4907-b60b-d77d89bffbba","Type":"ContainerStarted","Data":"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"}
Jul 30 21:27:08.829384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.828926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" event={"ID":"512943c9-5477-4907-b60b-d77d89bffbba","Type":"ContainerStarted","Data":"89b2a5865a9dee1f64ebe298c2cbf3bce6bb59dadd53eb1238fc7c3e34bd35c9"}
Jul 30 21:27:08.829384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.828980    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:08.830960 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.830921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" event={"ID":"171a9755-6e90-4f00-b28b-6bf2c8a2ee68","Type":"ContainerStarted","Data":"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"}
Jul 30 21:27:08.831101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.830975    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" event={"ID":"171a9755-6e90-4f00-b28b-6bf2c8a2ee68","Type":"ContainerStarted","Data":"407318a38fc05935cfd24f6fa65acd1a2c45f01e07af7937786f4735cea8caaf"}
Jul 30 21:27:08.831101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.831016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:08.842881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.842838    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" podStartSLOduration=1.842822379 podStartE2EDuration="1.842822379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:08.841705762 +0000 UTC m=+1800.607667642" watchObservedRunningTime="2026-07-30 21:27:08.842822379 +0000 UTC m=+1800.608784258"
Jul 30 21:27:08.857235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:08.857180    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" podStartSLOduration=1.857161954 podStartE2EDuration="1.857161954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:08.855069307 +0000 UTC m=+1800.621031187" watchObservedRunningTime="2026-07-30 21:27:08.857161954 +0000 UTC m=+1800.623123836"
Jul 30 21:27:09.108724 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:09.108654    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:27:09.110163 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:09.110139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:27:09.837577 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:09.837532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" event={"ID":"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4","Type":"ContainerStarted","Data":"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"}
Jul 30 21:27:09.837982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:09.837916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:09.853127 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:09.853077    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" podStartSLOduration=1.9233279699999999 podStartE2EDuration="3.853063881s" totalImagesPullingTime="1.929735911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:09.851242109 +0000 UTC m=+1801.617203987" watchObservedRunningTime="2026-07-30 21:27:09.853063881 +0000 UTC m=+1801.619025759"
Jul 30 21:27:15.848738 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:15.848707    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:19.839731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:19.839695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:19.840209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:19.839862    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:31.781602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.781548    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"]
Jul 30 21:27:31.785157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.785125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:31.787026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.787001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:27:31.792019 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.791991    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"]
Jul 30 21:27:31.797636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.797588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs\" (UID: \"9e3802d0-6257-4710-a2d3-08aa75077c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:31.898753 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.898721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs\" (UID: \"9e3802d0-6257-4710-a2d3-08aa75077c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:31.899342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:31.899321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs\" (UID: \"9e3802d0-6257-4710-a2d3-08aa75077c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:32.098390 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:32.098314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:32.235180 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:32.235111    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"]
Jul 30 21:27:32.237681 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:32.237652    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3802d0_6257_4710_a2d3_08aa75077c95.slice/crio-22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5 WatchSource:0}: Error finding container 22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5: Status 404 returned error can't find the container with id 22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5
Jul 30 21:27:32.944968 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:32.944929    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e3802d0-6257-4710-a2d3-08aa75077c95" containerID="c46d88fbaadf7f0620be59f8a5c57d5bac2b8eb3d7742c838460d4b4598ca134" exitCode=0
Jul 30 21:27:32.945321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:32.944978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs" event={"ID":"9e3802d0-6257-4710-a2d3-08aa75077c95","Type":"ContainerDied","Data":"c46d88fbaadf7f0620be59f8a5c57d5bac2b8eb3d7742c838460d4b4598ca134"}
Jul 30 21:27:32.945321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:32.945035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs" event={"ID":"9e3802d0-6257-4710-a2d3-08aa75077c95","Type":"ContainerStarted","Data":"22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5"}
Jul 30 21:27:34.089431 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.089405    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:34.116986 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.116956    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca\") pod \"9e3802d0-6257-4710-a2d3-08aa75077c95\" (UID: \"9e3802d0-6257-4710-a2d3-08aa75077c95\") "
Jul 30 21:27:34.117318 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.117292    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e3802d0-6257-4710-a2d3-08aa75077c95" (UID: "9e3802d0-6257-4710-a2d3-08aa75077c95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:34.217841 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.217758    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3802d0-6257-4710-a2d3-08aa75077c95-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:34.962059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.962023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs" event={"ID":"9e3802d0-6257-4710-a2d3-08aa75077c95","Type":"ContainerDied","Data":"22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5"}
Jul 30 21:27:34.962059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.962042    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"
Jul 30 21:27:34.962253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:34.962082    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22fab26b7e99724711411663aafd040a09d2553d16058303aa503c74fd8f5ba5"
Jul 30 21:27:35.865769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.865724    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:35.866185 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.865937    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" podUID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" containerName="maas-api" containerID="cri-o://ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f" gracePeriod=30
Jul 30 21:27:35.893078 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.893033    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:35.913419 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.913370    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:35.913653 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.913625    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" podUID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" containerName="payload-processing" containerID="cri-o://c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec" gracePeriod=30
Jul 30 21:27:35.921401 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.921369    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:35.921671 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.921639    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" podUID="512943c9-5477-4907-b60b-d77d89bffbba" containerName="payload-pre-processing" containerID="cri-o://356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f" gracePeriod=30
Jul 30 21:27:35.952260 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:35.952219    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.056867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.056757    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.056867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.056855    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"]
Jul 30 21:27:36.059895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.059514    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60d76360-m7vfs"]
Jul 30 21:27:36.154495 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.154440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.215884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.215853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:36.225756 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.225722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.226502 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.226485    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:36.232861 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.232839    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume\") pod \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") "
Jul 30 21:27:36.232989 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.232899    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkn6v\" (UniqueName: \"kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v\") pod \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\" (UID: \"171a9755-6e90-4f00-b28b-6bf2c8a2ee68\") "
Jul 30 21:27:36.233308 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.233281    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "171a9755-6e90-4f00-b28b-6bf2c8a2ee68" (UID: "171a9755-6e90-4f00-b28b-6bf2c8a2ee68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:36.237314 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.237283    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v" (OuterVolumeSpecName: "kube-api-access-tkn6v") pod "171a9755-6e90-4f00-b28b-6bf2c8a2ee68" (UID: "171a9755-6e90-4f00-b28b-6bf2c8a2ee68"). InnerVolumeSpecName "kube-api-access-tkn6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:27:36.239920 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.239899    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:36.333776 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.333742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume\") pod \"512943c9-5477-4907-b60b-d77d89bffbba\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") "
Jul 30 21:27:36.333965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.333802    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wrvk\" (UniqueName: \"kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk\") pod \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") "
Jul 30 21:27:36.333965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.333824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txrsw\" (UniqueName: \"kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw\") pod \"512943c9-5477-4907-b60b-d77d89bffbba\" (UID: \"512943c9-5477-4907-b60b-d77d89bffbba\") "
Jul 30 21:27:36.334060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334010    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls\") pod \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") "
Jul 30 21:27:36.334117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca\") pod \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\" (UID: \"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4\") "
Jul 30 21:27:36.334236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "512943c9-5477-4907-b60b-d77d89bffbba" (UID: "512943c9-5477-4907-b60b-d77d89bffbba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:36.334630 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334458    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" (UID: "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:27:36.334630 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334467    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512943c9-5477-4907-b60b-d77d89bffbba-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.334630 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334515    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.334630 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.334531    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkn6v\" (UniqueName: \"kubernetes.io/projected/171a9755-6e90-4f00-b28b-6bf2c8a2ee68-kube-api-access-tkn6v\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.336127 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.336100    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw" (OuterVolumeSpecName: "kube-api-access-txrsw") pod "512943c9-5477-4907-b60b-d77d89bffbba" (UID: "512943c9-5477-4907-b60b-d77d89bffbba"). InnerVolumeSpecName "kube-api-access-txrsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:27:36.336212 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.336187    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk" (OuterVolumeSpecName: "kube-api-access-7wrvk") pod "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" (UID: "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4"). InnerVolumeSpecName "kube-api-access-7wrvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:27:36.336259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.336228    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" (UID: "347c59fd-c9f3-45b7-a113-2e86d3c6e0c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:27:36.435515 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.435437    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrvk\" (UniqueName: \"kubernetes.io/projected/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-kube-api-access-7wrvk\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.435515 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.435471    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txrsw\" (UniqueName: \"kubernetes.io/projected/512943c9-5477-4907-b60b-d77d89bffbba-kube-api-access-txrsw\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.435515 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.435486    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.435515 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.435495    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:27:36.463157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.463125    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.716915 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.716821    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:36.777332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.777296    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e3802d0-6257-4710-a2d3-08aa75077c95" path="/var/lib/kubelet/pods/9e3802d0-6257-4710-a2d3-08aa75077c95/volumes"
Jul 30 21:27:36.974016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.973921    2583 generic.go:356] "Generic (PLEG): container finished" podID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" containerID="c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec" exitCode=0
Jul 30 21:27:36.974016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.973971    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"
Jul 30 21:27:36.974016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.973996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" event={"ID":"171a9755-6e90-4f00-b28b-6bf2c8a2ee68","Type":"ContainerDied","Data":"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"}
Jul 30 21:27:36.974666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.974057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd" event={"ID":"171a9755-6e90-4f00-b28b-6bf2c8a2ee68","Type":"ContainerDied","Data":"407318a38fc05935cfd24f6fa65acd1a2c45f01e07af7937786f4735cea8caaf"}
Jul 30 21:27:36.974666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.974083    2583 scope.go:117] "RemoveContainer" containerID="c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"
Jul 30 21:27:36.975765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.975731    2583 generic.go:356] "Generic (PLEG): container finished" podID="512943c9-5477-4907-b60b-d77d89bffbba" containerID="356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f" exitCode=0
Jul 30 21:27:36.975765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.975777    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"
Jul 30 21:27:36.975765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.975775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" event={"ID":"512943c9-5477-4907-b60b-d77d89bffbba","Type":"ContainerDied","Data":"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"}
Jul 30 21:27:36.975765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.975831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd" event={"ID":"512943c9-5477-4907-b60b-d77d89bffbba","Type":"ContainerDied","Data":"89b2a5865a9dee1f64ebe298c2cbf3bce6bb59dadd53eb1238fc7c3e34bd35c9"}
Jul 30 21:27:36.977526 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.977503    2583 generic.go:356] "Generic (PLEG): container finished" podID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" containerID="ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f" exitCode=0
Jul 30 21:27:36.977677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.977536    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" event={"ID":"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4","Type":"ContainerDied","Data":"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"}
Jul 30 21:27:36.977677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.977593    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"
Jul 30 21:27:36.977677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.977610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd" event={"ID":"347c59fd-c9f3-45b7-a113-2e86d3c6e0c4","Type":"ContainerDied","Data":"4a521858e5d2dedaf1d9466b591311cc745f7f6ce9d087edfb8db556fbfc12b7"}
Jul 30 21:27:36.984147 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.984122    2583 scope.go:117] "RemoveContainer" containerID="c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"
Jul 30 21:27:36.984512 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:27:36.984484    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec\": container with ID starting with c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec not found: ID does not exist" containerID="c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"
Jul 30 21:27:36.984688 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.984521    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec"} err="failed to get container status \"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec\": rpc error: code = NotFound desc = could not find container \"c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec\": container with ID starting with c7c437e6b3edacdc263257376a96e5698d11916a49e1653d641bc6e52b979fec not found: ID does not exist"
Jul 30 21:27:36.984688 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.984545    2583 scope.go:117] "RemoveContainer" containerID="356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"
Jul 30 21:27:36.994947 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.994917    2583 scope.go:117] "RemoveContainer" containerID="356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"
Jul 30 21:27:36.995312 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:27:36.995282    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f\": container with ID starting with 356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f not found: ID does not exist" containerID="356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"
Jul 30 21:27:36.995433 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.995317    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f"} err="failed to get container status \"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f\": rpc error: code = NotFound desc = could not find container \"356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f\": container with ID starting with 356dec769d4a0c3ae3687ffa3a4c19091e16b9e6676ba9f4f279315678a1a01f not found: ID does not exist"
Jul 30 21:27:36.995433 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:36.995337    2583 scope.go:117] "RemoveContainer" containerID="ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"
Jul 30 21:27:37.003557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.003516    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:37.005995 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.005471    2583 scope.go:117] "RemoveContainer" containerID="ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"
Jul 30 21:27:37.005995 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:27:37.005963    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f\": container with ID starting with ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f not found: ID does not exist" containerID="ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"
Jul 30 21:27:37.006129 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.006006    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f"} err="failed to get container status \"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f\": rpc error: code = NotFound desc = could not find container \"ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f\": container with ID starting with ef213b82d11c9a458bce1a4a11b16bfb38bdae01c4570a1c7ee289707f62b38f not found: ID does not exist"
Jul 30 21:27:37.019122 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.013308    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60d76360-8677878cc-7m8nd"]
Jul 30 21:27:37.032774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.032743    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:37.041492 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.041451    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-60d76360-7c5d499dfc-njchd"]
Jul 30 21:27:37.052652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.052613    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:37.058058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.058007    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-60d76360-6fb6d66ff6-ddmzd"]
Jul 30 21:27:37.288502 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.288464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:37.337240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:37.337201    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:38.777362 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:38.777320    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" path="/var/lib/kubelet/pods/171a9755-6e90-4f00-b28b-6bf2c8a2ee68/volumes"
Jul 30 21:27:38.777913 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:38.777890    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" path="/var/lib/kubelet/pods/347c59fd-c9f3-45b7-a113-2e86d3c6e0c4/volumes"
Jul 30 21:27:38.778251 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:38.778233    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512943c9-5477-4907-b60b-d77d89bffbba" path="/var/lib/kubelet/pods/512943c9-5477-4907-b60b-d77d89bffbba/volumes"
Jul 30 21:27:39.694376 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:39.694336    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:39.792002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:39.791936    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:40.796287 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796247    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" containerName="maas-api"
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796644    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" containerName="maas-api"
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796653    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512943c9-5477-4907-b60b-d77d89bffbba" containerName="payload-pre-processing"
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796659    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="512943c9-5477-4907-b60b-d77d89bffbba" containerName="payload-pre-processing"
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796674    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" containerName="payload-processing"
Jul 30 21:27:40.796676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796680    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" containerName="payload-processing"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796690    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3802d0-6257-4710-a2d3-08aa75077c95" containerName="revoke-keys"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796696    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3802d0-6257-4710-a2d3-08aa75077c95" containerName="revoke-keys"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796745    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="512943c9-5477-4907-b60b-d77d89bffbba" containerName="payload-pre-processing"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796756    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="171a9755-6e90-4f00-b28b-6bf2c8a2ee68" containerName="payload-processing"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796761    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="347c59fd-c9f3-45b7-a113-2e86d3c6e0c4" containerName="maas-api"
Jul 30 21:27:40.796869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.796766    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e3802d0-6257-4710-a2d3-08aa75077c95" containerName="revoke-keys"
Jul 30 21:27:40.799708 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.799688    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.801681 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.801658    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-55614fcd-serving-cert\""
Jul 30 21:27:40.801789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.801709    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:27:40.807438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.807414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:27:40.836551 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.836517    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:40.869468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.869432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8w6\" (UniqueName: \"kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.869665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.869486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.869665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.869553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.939281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.939241    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:40.970355 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.970322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8w6\" (UniqueName: \"kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.970479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.970409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.970479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.970456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.971108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.971086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.973118 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.973097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:40.977685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:40.977663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8w6\" (UniqueName: \"kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6\") pod \"maas-api-e2e-ait-55614fcd-74f784bc98-mzx55\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:41.113231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.113149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:41.255705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.255672    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:27:41.259006 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:41.258976    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc13801_311c_4b78_b9fc_ad95ac250755.slice/crio-4b3eee8bc36b743271f7c4cff02a9e002b001d3ce743fd755b16ede9f0c56da8 WatchSource:0}: Error finding container 4b3eee8bc36b743271f7c4cff02a9e002b001d3ce743fd755b16ede9f0c56da8: Status 404 returned error can't find the container with id 4b3eee8bc36b743271f7c4cff02a9e002b001d3ce743fd755b16ede9f0c56da8
Jul 30 21:27:41.340145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.340103    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:41.439899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.439799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:27:41.456897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.456869    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:27:41.461253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.461231    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.463057 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.463041    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-55614fcd-dockercfg-fx895\""
Jul 30 21:27:41.463103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.463077    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-55614fcd\""
Jul 30 21:27:41.469923 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.469902    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:27:41.507603 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.507556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:27:41.510794 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.510776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.522299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.522263    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:27:41.576910 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.576876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9tx\" (UniqueName: \"kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.577075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.576941    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.577075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.577031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.577162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.577080    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m425t\" (UniqueName: \"kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.677547 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.677511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.677652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.677579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.677652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.677602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m425t\" (UniqueName: \"kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.677652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.677641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9tx\" (UniqueName: \"kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.678417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.678392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.678468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.678391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.685495 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.685458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m425t\" (UniqueName: \"kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t\") pod \"payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.687904 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.687878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9tx\" (UniqueName: \"kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx\") pod \"payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") " pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.773111 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.773077    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:41.828005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.827423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:41.924861 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.924829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:27:41.980409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:41.980378    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:27:41.983527 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:27:41.983500    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2458fc3b_4e40_43c3_860c_ad34c80b4078.slice/crio-c83b868f8c6f9de326372c8c23309017ef1ba44c4d7bd66c741715130c472525 WatchSource:0}: Error finding container c83b868f8c6f9de326372c8c23309017ef1ba44c4d7bd66c741715130c472525: Status 404 returned error can't find the container with id c83b868f8c6f9de326372c8c23309017ef1ba44c4d7bd66c741715130c472525
Jul 30 21:27:42.010086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.010055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" event={"ID":"d674178d-5fa4-41c7-96ed-21652c507261","Type":"ContainerStarted","Data":"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"}
Jul 30 21:27:42.010215 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.010125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" event={"ID":"d674178d-5fa4-41c7-96ed-21652c507261","Type":"ContainerStarted","Data":"238e4fd00861870cbacdcc21dc44a4464248af1b8b44a74753cb0cdbc46c81af"}
Jul 30 21:27:42.010274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.010243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:42.011835 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.011787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" event={"ID":"5dc13801-311c-4b78-b9fc-ad95ac250755","Type":"ContainerStarted","Data":"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"}
Jul 30 21:27:42.011932 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.011838    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:42.011932 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.011862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" event={"ID":"5dc13801-311c-4b78-b9fc-ad95ac250755","Type":"ContainerStarted","Data":"4b3eee8bc36b743271f7c4cff02a9e002b001d3ce743fd755b16ede9f0c56da8"}
Jul 30 21:27:42.013225 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.013201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" event={"ID":"2458fc3b-4e40-43c3-860c-ad34c80b4078","Type":"ContainerStarted","Data":"c83b868f8c6f9de326372c8c23309017ef1ba44c4d7bd66c741715130c472525"}
Jul 30 21:27:42.025047 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.024626    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" podStartSLOduration=1.024611713 podStartE2EDuration="1.024611713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:42.024232828 +0000 UTC m=+1833.790194707" watchObservedRunningTime="2026-07-30 21:27:42.024611713 +0000 UTC m=+1833.790573586"
Jul 30 21:27:42.040186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:42.040138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" podStartSLOduration=1.6424838130000001 podStartE2EDuration="2.040123208s" totalImagesPullingTime="397.639395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:42.038534922 +0000 UTC m=+1833.804496802" watchObservedRunningTime="2026-07-30 21:27:42.040123208 +0000 UTC m=+1833.806085087"
Jul 30 21:27:43.024111 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:43.024069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" event={"ID":"2458fc3b-4e40-43c3-860c-ad34c80b4078","Type":"ContainerStarted","Data":"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"}
Jul 30 21:27:43.024595 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:43.024220    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:27:43.036881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:43.036825    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" podStartSLOduration=2.036806329 podStartE2EDuration="2.036806329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:27:43.036555972 +0000 UTC m=+1834.802517852" watchObservedRunningTime="2026-07-30 21:27:43.036806329 +0000 UTC m=+1834.802768200"
Jul 30 21:27:48.029265 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:48.029236    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:27:53.026640 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:53.026610    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:27:54.032774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:27:54.032740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:28:10.814336 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:10.814300    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"]
Jul 30 21:28:10.821824 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:10.821790    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:10.824386 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:10.824361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:28:10.825335 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:10.825312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"]
Jul 30 21:28:10.932427 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:10.932378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4\" (UID: \"2a057131-3ac9-4ad5-b11e-cf3f212f845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:11.033615 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:11.033550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4\" (UID: \"2a057131-3ac9-4ad5-b11e-cf3f212f845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:11.034241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:11.034218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4\" (UID: \"2a057131-3ac9-4ad5-b11e-cf3f212f845e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:11.136044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:11.135958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:11.271724 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:11.271697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"]
Jul 30 21:28:11.274184 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:28:11.274156    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a057131_3ac9_4ad5_b11e_cf3f212f845e.slice/crio-365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454 WatchSource:0}: Error finding container 365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454: Status 404 returned error can't find the container with id 365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454
Jul 30 21:28:12.169014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:12.168970    2583 generic.go:356] "Generic (PLEG): container finished" podID="2a057131-3ac9-4ad5-b11e-cf3f212f845e" containerID="ce050eeef6b01c035d386ec72bc618f1344afe2bf57d7ecc0b498574f8a08c49" exitCode=0
Jul 30 21:28:12.169429 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:12.169061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4" event={"ID":"2a057131-3ac9-4ad5-b11e-cf3f212f845e","Type":"ContainerDied","Data":"ce050eeef6b01c035d386ec72bc618f1344afe2bf57d7ecc0b498574f8a08c49"}
Jul 30 21:28:12.169429 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:12.169131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4" event={"ID":"2a057131-3ac9-4ad5-b11e-cf3f212f845e","Type":"ContainerStarted","Data":"365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454"}
Jul 30 21:28:13.310138 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:13.310114    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:13.454786 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:13.454706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca\") pod \"2a057131-3ac9-4ad5-b11e-cf3f212f845e\" (UID: \"2a057131-3ac9-4ad5-b11e-cf3f212f845e\") "
Jul 30 21:28:13.455033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:13.455010    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a057131-3ac9-4ad5-b11e-cf3f212f845e" (UID: "2a057131-3ac9-4ad5-b11e-cf3f212f845e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:13.555528 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:13.555493    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a057131-3ac9-4ad5-b11e-cf3f212f845e-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:14.180748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:14.180714    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"
Jul 30 21:28:14.180931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:14.180717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4" event={"ID":"2a057131-3ac9-4ad5-b11e-cf3f212f845e","Type":"ContainerDied","Data":"365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454"}
Jul 30 21:28:14.180931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:14.180852    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="365cb19e2219abbe8c8890252b521b25b760cad318269847357492ced8837454"
Jul 30 21:28:21.367692 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.367655    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:28:21.368514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.368453    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" podUID="5dc13801-311c-4b78-b9fc-ad95ac250755" containerName="maas-api" containerID="cri-o://8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1" gracePeriod=30
Jul 30 21:28:21.388305 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.388255    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:21.422971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.422938    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:28:21.423175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.423149    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" podUID="d674178d-5fa4-41c7-96ed-21652c507261" containerName="payload-processing" containerID="cri-o://53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499" gracePeriod=30
Jul 30 21:28:21.430060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.430028    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:28:21.430264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.430232    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" podUID="2458fc3b-4e40-43c3-860c-ad34c80b4078" containerName="payload-pre-processing" containerID="cri-o://933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f" gracePeriod=30
Jul 30 21:28:21.463830 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.463798    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:21.570509 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.569668    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"]
Jul 30 21:28:21.572939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.572790    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-55614fcd-dn7v4"]
Jul 30 21:28:21.718623 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.718595    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:28:21.721143 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.721122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz9tx\" (UniqueName: \"kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx\") pod \"d674178d-5fa4-41c7-96ed-21652c507261\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") "
Jul 30 21:28:21.721235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.721205    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume\") pod \"d674178d-5fa4-41c7-96ed-21652c507261\" (UID: \"d674178d-5fa4-41c7-96ed-21652c507261\") "
Jul 30 21:28:21.721627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.721603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d674178d-5fa4-41c7-96ed-21652c507261" (UID: "d674178d-5fa4-41c7-96ed-21652c507261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:21.723494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.723470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx" (OuterVolumeSpecName: "kube-api-access-bz9tx") pod "d674178d-5fa4-41c7-96ed-21652c507261" (UID: "d674178d-5fa4-41c7-96ed-21652c507261"). InnerVolumeSpecName "kube-api-access-bz9tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:21.728329 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.728308    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:28:21.742132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.742101    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:28:21.822098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822059    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls\") pod \"5dc13801-311c-4b78-b9fc-ad95ac250755\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") "
Jul 30 21:28:21.822098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume\") pod \"2458fc3b-4e40-43c3-860c-ad34c80b4078\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") "
Jul 30 21:28:21.822350 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca\") pod \"5dc13801-311c-4b78-b9fc-ad95ac250755\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") "
Jul 30 21:28:21.822350 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822148    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m425t\" (UniqueName: \"kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t\") pod \"2458fc3b-4e40-43c3-860c-ad34c80b4078\" (UID: \"2458fc3b-4e40-43c3-860c-ad34c80b4078\") "
Jul 30 21:28:21.822350 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822342    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz9tx\" (UniqueName: \"kubernetes.io/projected/d674178d-5fa4-41c7-96ed-21652c507261-kube-api-access-bz9tx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.822514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822360    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d674178d-5fa4-41c7-96ed-21652c507261-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.822514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dc13801-311c-4b78-b9fc-ad95ac250755" (UID: "5dc13801-311c-4b78-b9fc-ad95ac250755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:21.822514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.822464    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2458fc3b-4e40-43c3-860c-ad34c80b4078" (UID: "2458fc3b-4e40-43c3-860c-ad34c80b4078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:21.824125 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.824097    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dc13801-311c-4b78-b9fc-ad95ac250755" (UID: "5dc13801-311c-4b78-b9fc-ad95ac250755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:28:21.824315 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.824290    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t" (OuterVolumeSpecName: "kube-api-access-m425t") pod "2458fc3b-4e40-43c3-860c-ad34c80b4078" (UID: "2458fc3b-4e40-43c3-860c-ad34c80b4078"). InnerVolumeSpecName "kube-api-access-m425t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:21.923221 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.923131    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8w6\" (UniqueName: \"kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6\") pod \"5dc13801-311c-4b78-b9fc-ad95ac250755\" (UID: \"5dc13801-311c-4b78-b9fc-ad95ac250755\") "
Jul 30 21:28:21.923360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.923263    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc13801-311c-4b78-b9fc-ad95ac250755-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.923360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.923274    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2458fc3b-4e40-43c3-860c-ad34c80b4078-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.923360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.923284    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc13801-311c-4b78-b9fc-ad95ac250755-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.923360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.923293    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m425t\" (UniqueName: \"kubernetes.io/projected/2458fc3b-4e40-43c3-860c-ad34c80b4078-kube-api-access-m425t\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:21.925292 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:21.925267    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6" (OuterVolumeSpecName: "kube-api-access-7c8w6") pod "5dc13801-311c-4b78-b9fc-ad95ac250755" (UID: "5dc13801-311c-4b78-b9fc-ad95ac250755"). InnerVolumeSpecName "kube-api-access-7c8w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:22.023763 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.023722    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8w6\" (UniqueName: \"kubernetes.io/projected/5dc13801-311c-4b78-b9fc-ad95ac250755-kube-api-access-7c8w6\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:22.218426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.218336    2583 generic.go:356] "Generic (PLEG): container finished" podID="d674178d-5fa4-41c7-96ed-21652c507261" containerID="53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499" exitCode=0
Jul 30 21:28:22.218426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.218388    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"
Jul 30 21:28:22.218683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.218418    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" event={"ID":"d674178d-5fa4-41c7-96ed-21652c507261","Type":"ContainerDied","Data":"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"}
Jul 30 21:28:22.218683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.218477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg" event={"ID":"d674178d-5fa4-41c7-96ed-21652c507261","Type":"ContainerDied","Data":"238e4fd00861870cbacdcc21dc44a4464248af1b8b44a74753cb0cdbc46c81af"}
Jul 30 21:28:22.218683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.218504    2583 scope.go:117] "RemoveContainer" containerID="53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"
Jul 30 21:28:22.219868 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.219846    2583 generic.go:356] "Generic (PLEG): container finished" podID="5dc13801-311c-4b78-b9fc-ad95ac250755" containerID="8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1" exitCode=0
Jul 30 21:28:22.219964 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.219882    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"
Jul 30 21:28:22.219964 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.219937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" event={"ID":"5dc13801-311c-4b78-b9fc-ad95ac250755","Type":"ContainerDied","Data":"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"}
Jul 30 21:28:22.220132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.219986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55" event={"ID":"5dc13801-311c-4b78-b9fc-ad95ac250755","Type":"ContainerDied","Data":"4b3eee8bc36b743271f7c4cff02a9e002b001d3ce743fd755b16ede9f0c56da8"}
Jul 30 21:28:22.221725 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.221587    2583 generic.go:356] "Generic (PLEG): container finished" podID="2458fc3b-4e40-43c3-860c-ad34c80b4078" containerID="933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f" exitCode=0
Jul 30 21:28:22.221957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.221712    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" event={"ID":"2458fc3b-4e40-43c3-860c-ad34c80b4078","Type":"ContainerDied","Data":"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"}
Jul 30 21:28:22.221957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.221765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q" event={"ID":"2458fc3b-4e40-43c3-860c-ad34c80b4078","Type":"ContainerDied","Data":"c83b868f8c6f9de326372c8c23309017ef1ba44c4d7bd66c741715130c472525"}
Jul 30 21:28:22.221957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.221827    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"
Jul 30 21:28:22.232217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.232066    2583 scope.go:117] "RemoveContainer" containerID="53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"
Jul 30 21:28:22.232636 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:22.232493    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499\": container with ID starting with 53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499 not found: ID does not exist" containerID="53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"
Jul 30 21:28:22.232636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.232532    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499"} err="failed to get container status \"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499\": rpc error: code = NotFound desc = could not find container \"53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499\": container with ID starting with 53e98261aa0571cfeb027312cecee198c12ae3f13c3636bc74e3225f26fe6499 not found: ID does not exist"
Jul 30 21:28:22.232636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.232557    2583 scope.go:117] "RemoveContainer" containerID="8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"
Jul 30 21:28:22.241158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.241141    2583 scope.go:117] "RemoveContainer" containerID="8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"
Jul 30 21:28:22.241493 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:22.241461    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1\": container with ID starting with 8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1 not found: ID does not exist" containerID="8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"
Jul 30 21:28:22.241607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.241498    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1"} err="failed to get container status \"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1\": rpc error: code = NotFound desc = could not find container \"8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1\": container with ID starting with 8716a6f1f2b7ab05ae495e9ef6fe66881c6b7763931b98cac8d67f0bbfaeebe1 not found: ID does not exist"
Jul 30 21:28:22.241607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.241520    2583 scope.go:117] "RemoveContainer" containerID="933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"
Jul 30 21:28:22.244213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.244153    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:28:22.245840 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.245822    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-55614fcd-d7f67bc84-5cjsg"]
Jul 30 21:28:22.249926 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.249911    2583 scope.go:117] "RemoveContainer" containerID="933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"
Jul 30 21:28:22.250233 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:22.250216    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f\": container with ID starting with 933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f not found: ID does not exist" containerID="933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"
Jul 30 21:28:22.250271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.250241    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f"} err="failed to get container status \"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f\": rpc error: code = NotFound desc = could not find container \"933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f\": container with ID starting with 933006f75edd1b015e4f5228d4a8ae5d8a65383de157bcf846beede369a3749f not found: ID does not exist"
Jul 30 21:28:22.258242 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.257836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:28:22.265694 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.265674    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-55614fcd-74f784bc98-mzx55"]
Jul 30 21:28:22.274275 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.274250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:28:22.278253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.278232    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55614fcd-6879869f56-t7t5q"]
Jul 30 21:28:22.397112 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.397073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:22.497895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.497862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:22.777191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.777112    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2458fc3b-4e40-43c3-860c-ad34c80b4078" path="/var/lib/kubelet/pods/2458fc3b-4e40-43c3-860c-ad34c80b4078/volumes"
Jul 30 21:28:22.777417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.777405    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a057131-3ac9-4ad5-b11e-cf3f212f845e" path="/var/lib/kubelet/pods/2a057131-3ac9-4ad5-b11e-cf3f212f845e/volumes"
Jul 30 21:28:22.777765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.777750    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc13801-311c-4b78-b9fc-ad95ac250755" path="/var/lib/kubelet/pods/5dc13801-311c-4b78-b9fc-ad95ac250755/volumes"
Jul 30 21:28:22.778044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.778032    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d674178d-5fa4-41c7-96ed-21652c507261" path="/var/lib/kubelet/pods/d674178d-5fa4-41c7-96ed-21652c507261/volumes"
Jul 30 21:28:22.860490 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:22.860454    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:23.106848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:23.106764    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:23.361758 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:23.361673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:23.774937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:23.774891    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:23.947314 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:23.947276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:24.047627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:24.047518    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:24.188788 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:24.188745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:25.960268 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:25.960190    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:26.061153 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:26.061107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:27.076065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076029    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076374    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2458fc3b-4e40-43c3-860c-ad34c80b4078" containerName="payload-pre-processing"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076391    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2458fc3b-4e40-43c3-860c-ad34c80b4078" containerName="payload-pre-processing"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076404    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a057131-3ac9-4ad5-b11e-cf3f212f845e" containerName="revoke-keys"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076412    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a057131-3ac9-4ad5-b11e-cf3f212f845e" containerName="revoke-keys"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076427    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674178d-5fa4-41c7-96ed-21652c507261" containerName="payload-processing"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076436    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674178d-5fa4-41c7-96ed-21652c507261" containerName="payload-processing"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076450    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc13801-311c-4b78-b9fc-ad95ac250755" containerName="maas-api"
Jul 30 21:28:27.076475 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076458    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc13801-311c-4b78-b9fc-ad95ac250755" containerName="maas-api"
Jul 30 21:28:27.076982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076559    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2458fc3b-4e40-43c3-860c-ad34c80b4078" containerName="payload-pre-processing"
Jul 30 21:28:27.076982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076628    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc13801-311c-4b78-b9fc-ad95ac250755" containerName="maas-api"
Jul 30 21:28:27.076982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076636    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d674178d-5fa4-41c7-96ed-21652c507261" containerName="payload-processing"
Jul 30 21:28:27.076982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.076645    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a057131-3ac9-4ad5-b11e-cf3f212f845e" containerName="revoke-keys"
Jul 30 21:28:27.081347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.081326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.085818 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.085794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b3059c6e-serving-cert\""
Jul 30 21:28:27.086160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.086095    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:28:27.093001 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.092971    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:27.114527 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.114501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:27.166612 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.166554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.166792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.166651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fckx\" (UniqueName: \"kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.166792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.166713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.268113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.268083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fckx\" (UniqueName: \"kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.268291 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.268134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.268291 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.268172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.268371 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:27.268308    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-serving-cert: secret "maas-api-e2e-derive-b3059c6e-serving-cert" not found
Jul 30 21:28:27.268406 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:27.268374    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls podName:bcb4a170-bf04-4211-a41b-a3e9745c8c03 nodeName:}" failed. No retries permitted until 2026-07-30 21:28:27.768353668 +0000 UTC m=+1879.534315524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls") pod "maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" (UID: "bcb4a170-bf04-4211-a41b-a3e9745c8c03") : secret "maas-api-e2e-derive-b3059c6e-serving-cert" not found
Jul 30 21:28:27.268807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.268790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.276153 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.276131    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fckx\" (UniqueName: \"kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.591231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.591184    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:27.702188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.702155    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:27.702734 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.702713    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:27.706186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.706167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.708046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.708019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b3059c6e-dockercfg-mg4cg\""
Jul 30 21:28:27.708186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.708167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b3059c6e\""
Jul 30 21:28:27.716031 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.715996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:27.730369 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.730347    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:27.737995 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.737971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.741690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.741656    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:27.771641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.771614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.771769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.771663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx2v\" (UniqueName: \"kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.771769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.771699    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9xb\" (UniqueName: \"kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.771769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.771724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.771876 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.771809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.774038 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.774012    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") pod \"maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:27.872591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.872477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx2v\" (UniqueName: \"kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.872591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.872531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9xb\" (UniqueName: \"kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.872591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.872556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.872815 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.872626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.873220 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.873186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.873261 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.873244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.880052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.880029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx2v\" (UniqueName: \"kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v\") pod \"payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:27.880282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.880256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9xb\" (UniqueName: \"kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb\") pod \"payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") " pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:27.992717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:27.992684    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:28.018157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.018132    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:28.050477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.050436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:28.160284 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.160248    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:28.163510 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:28:28.163476    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb4a170_bf04_4211_a41b_a3e9745c8c03.slice/crio-e4750711e2818da5e1bfcf6c2f818b9dc6d5eb483b6b2aa7f62d3e2a9fbe3a95 WatchSource:0}: Error finding container e4750711e2818da5e1bfcf6c2f818b9dc6d5eb483b6b2aa7f62d3e2a9fbe3a95: Status 404 returned error can't find the container with id e4750711e2818da5e1bfcf6c2f818b9dc6d5eb483b6b2aa7f62d3e2a9fbe3a95
Jul 30 21:28:28.186258 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.186128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:28.189305 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:28:28.189244    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f38dcf_630d_4159_9876_9daa4612d5b5.slice/crio-1917080a88bd34bc18393e4a184a3927871314b804c0ac0fb6dec4c284cb5802 WatchSource:0}: Error finding container 1917080a88bd34bc18393e4a184a3927871314b804c0ac0fb6dec4c284cb5802: Status 404 returned error can't find the container with id 1917080a88bd34bc18393e4a184a3927871314b804c0ac0fb6dec4c284cb5802
Jul 30 21:28:28.218251 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.218230    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:28.225701 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:28:28.225675    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339c6a6b_b8d5_4695_b339_db0bdac8d17b.slice/crio-b595396663ceb0d8df1ca752adcf398e5ddd029ae8fdfd7d629f96849b9fa73e WatchSource:0}: Error finding container b595396663ceb0d8df1ca752adcf398e5ddd029ae8fdfd7d629f96849b9fa73e: Status 404 returned error can't find the container with id b595396663ceb0d8df1ca752adcf398e5ddd029ae8fdfd7d629f96849b9fa73e
Jul 30 21:28:28.257113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.257081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" event={"ID":"339c6a6b-b8d5-4695-b339-db0bdac8d17b","Type":"ContainerStarted","Data":"b595396663ceb0d8df1ca752adcf398e5ddd029ae8fdfd7d629f96849b9fa73e"}
Jul 30 21:28:28.258733 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.258663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" event={"ID":"31f38dcf-630d-4159-9876-9daa4612d5b5","Type":"ContainerStarted","Data":"1917080a88bd34bc18393e4a184a3927871314b804c0ac0fb6dec4c284cb5802"}
Jul 30 21:28:28.259896 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:28.259874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" event={"ID":"bcb4a170-bf04-4211-a41b-a3e9745c8c03","Type":"ContainerStarted","Data":"e4750711e2818da5e1bfcf6c2f818b9dc6d5eb483b6b2aa7f62d3e2a9fbe3a95"}
Jul 30 21:28:29.266739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.266698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" event={"ID":"339c6a6b-b8d5-4695-b339-db0bdac8d17b","Type":"ContainerStarted","Data":"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"}
Jul 30 21:28:29.267248 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.266811    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:29.268209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.268186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" event={"ID":"31f38dcf-630d-4159-9876-9daa4612d5b5","Type":"ContainerStarted","Data":"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"}
Jul 30 21:28:29.268332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.268239    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:29.269850 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.269828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" event={"ID":"bcb4a170-bf04-4211-a41b-a3e9745c8c03","Type":"ContainerStarted","Data":"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"}
Jul 30 21:28:29.269961 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.269874    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:29.281216 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.281175    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" podStartSLOduration=2.281164019 podStartE2EDuration="2.281164019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:28:29.279587403 +0000 UTC m=+1881.045549279" watchObservedRunningTime="2026-07-30 21:28:29.281164019 +0000 UTC m=+1881.047125875"
Jul 30 21:28:29.294080 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.294021    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" podStartSLOduration=2.294004819 podStartE2EDuration="2.294004819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:28:29.293126529 +0000 UTC m=+1881.059088407" watchObservedRunningTime="2026-07-30 21:28:29.294004819 +0000 UTC m=+1881.059966701"
Jul 30 21:28:29.307598 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:29.307526    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" podStartSLOduration=1.949351789 podStartE2EDuration="2.307505279s" totalImagesPullingTime="358.15349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:28:29.306914369 +0000 UTC m=+1881.072876253" watchObservedRunningTime="2026-07-30 21:28:29.307505279 +0000 UTC m=+1881.073467160"
Jul 30 21:28:35.280421 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:35.280338    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:40.276983 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:40.276949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:40.277406 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:40.277130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:50.908163 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:50.908123    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"]
Jul 30 21:28:50.911580 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:50.911545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:50.913306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:50.913288    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:28:50.919044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:50.919017    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"]
Jul 30 21:28:50.968389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:50.968357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd\" (UID: \"acf461db-dd88-4911-aa56-988015407552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:51.068788 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:51.068749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd\" (UID: \"acf461db-dd88-4911-aa56-988015407552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:51.069451 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:51.069429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd\" (UID: \"acf461db-dd88-4911-aa56-988015407552\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:51.224865 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:51.224834    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:51.569091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:51.569069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"]
Jul 30 21:28:51.571767 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:28:51.571740    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf461db_dd88_4911_aa56_988015407552.slice/crio-1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14 WatchSource:0}: Error finding container 1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14: Status 404 returned error can't find the container with id 1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14
Jul 30 21:28:52.383268 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:52.383228    2583 generic.go:356] "Generic (PLEG): container finished" podID="acf461db-dd88-4911-aa56-988015407552" containerID="dde6333283b43ad497a23f0a5952f9b22ac8d95924dfb6ecebea63acd33be36a" exitCode=0
Jul 30 21:28:52.383660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:52.383308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd" event={"ID":"acf461db-dd88-4911-aa56-988015407552","Type":"ContainerDied","Data":"dde6333283b43ad497a23f0a5952f9b22ac8d95924dfb6ecebea63acd33be36a"}
Jul 30 21:28:52.383660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:52.383358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd" event={"ID":"acf461db-dd88-4911-aa56-988015407552","Type":"ContainerStarted","Data":"1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14"}
Jul 30 21:28:53.535881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:53.535844    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:53.591862 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:53.591821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca\") pod \"acf461db-dd88-4911-aa56-988015407552\" (UID: \"acf461db-dd88-4911-aa56-988015407552\") "
Jul 30 21:28:53.592203 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:53.592175    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acf461db-dd88-4911-aa56-988015407552" (UID: "acf461db-dd88-4911-aa56-988015407552"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:53.693148 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:53.693046    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf461db-dd88-4911-aa56-988015407552-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:54.394346 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:54.394314    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd" event={"ID":"acf461db-dd88-4911-aa56-988015407552","Type":"ContainerDied","Data":"1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14"}
Jul 30 21:28:54.394346 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:54.394330    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"
Jul 30 21:28:54.394557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:54.394360    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b7ca59e106abf11eadde197e90eb65c96d13a22fd486db8e62f2f487b2d2b14"
Jul 30 21:28:57.567159 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.567116    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:57.567821 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.567413    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" podUID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" containerName="maas-api" containerID="cri-o://1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361" gracePeriod=30
Jul 30 21:28:57.592936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.592902    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:57.620299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.620266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:57.620583 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.620534    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" podUID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" containerName="payload-processing" containerID="cri-o://3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5" gracePeriod=30
Jul 30 21:28:57.627749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.627723    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:57.627925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.627905    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" podUID="31f38dcf-630d-4159-9876-9daa4612d5b5" containerName="payload-pre-processing" containerID="cri-o://2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74" gracePeriod=30
Jul 30 21:28:57.655271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.655239    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:57.721323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.721283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:57.759809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.759365    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"]
Jul 30 21:28:57.761817 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.761790    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3059c6e-b6pcd"]
Jul 30 21:28:57.784117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.784038    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:57.913504 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.913478    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:57.914188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.914100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:57.931907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.931884    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:57.946426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:57.946400    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:58.033321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033282    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume\") pod \"31f38dcf-630d-4159-9876-9daa4612d5b5\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") "
Jul 30 21:28:58.033321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxx2v\" (UniqueName: \"kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v\") pod \"31f38dcf-630d-4159-9876-9daa4612d5b5\" (UID: \"31f38dcf-630d-4159-9876-9daa4612d5b5\") "
Jul 30 21:28:58.033602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033362    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") pod \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") "
Jul 30 21:28:58.033602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fckx\" (UniqueName: \"kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx\") pod \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") "
Jul 30 21:28:58.033602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033430    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca\") pod \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\" (UID: \"bcb4a170-bf04-4211-a41b-a3e9745c8c03\") "
Jul 30 21:28:58.033602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume\") pod \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") "
Jul 30 21:28:58.033602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk9xb\" (UniqueName: \"kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb\") pod \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\" (UID: \"339c6a6b-b8d5-4695-b339-db0bdac8d17b\") "
Jul 30 21:28:58.033869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033838    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31f38dcf-630d-4159-9876-9daa4612d5b5" (UID: "31f38dcf-630d-4159-9876-9daa4612d5b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:58.034024 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033892    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcb4a170-bf04-4211-a41b-a3e9745c8c03" (UID: "bcb4a170-bf04-4211-a41b-a3e9745c8c03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:58.034024 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.033944    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "339c6a6b-b8d5-4695-b339-db0bdac8d17b" (UID: "339c6a6b-b8d5-4695-b339-db0bdac8d17b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:28:58.035832 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.035802    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb" (OuterVolumeSpecName: "kube-api-access-tk9xb") pod "339c6a6b-b8d5-4695-b339-db0bdac8d17b" (UID: "339c6a6b-b8d5-4695-b339-db0bdac8d17b"). InnerVolumeSpecName "kube-api-access-tk9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:58.035929 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.035807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx" (OuterVolumeSpecName: "kube-api-access-6fckx") pod "bcb4a170-bf04-4211-a41b-a3e9745c8c03" (UID: "bcb4a170-bf04-4211-a41b-a3e9745c8c03"). InnerVolumeSpecName "kube-api-access-6fckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:58.036152 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.036135    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v" (OuterVolumeSpecName: "kube-api-access-wxx2v") pod "31f38dcf-630d-4159-9876-9daa4612d5b5" (UID: "31f38dcf-630d-4159-9876-9daa4612d5b5"). InnerVolumeSpecName "kube-api-access-wxx2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:28:58.036152 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.036137    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcb4a170-bf04-4211-a41b-a3e9745c8c03" (UID: "bcb4a170-bf04-4211-a41b-a3e9745c8c03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134238    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcb4a170-bf04-4211-a41b-a3e9745c8c03-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134269    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/339c6a6b-b8d5-4695-b339-db0bdac8d17b-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134279    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9xb\" (UniqueName: \"kubernetes.io/projected/339c6a6b-b8d5-4695-b339-db0bdac8d17b-kube-api-access-tk9xb\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134289    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f38dcf-630d-4159-9876-9daa4612d5b5-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134298    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx2v\" (UniqueName: \"kubernetes.io/projected/31f38dcf-630d-4159-9876-9daa4612d5b5-kube-api-access-wxx2v\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134306    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcb4a170-bf04-4211-a41b-a3e9745c8c03-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.134326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.134314    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fckx\" (UniqueName: \"kubernetes.io/projected/bcb4a170-bf04-4211-a41b-a3e9745c8c03-kube-api-access-6fckx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:28:58.196000 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.195934    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:58.415294 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.415206    2583 generic.go:356] "Generic (PLEG): container finished" podID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" containerID="3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5" exitCode=0
Jul 30 21:28:58.415294 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.415253    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"
Jul 30 21:28:58.415500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.415290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" event={"ID":"339c6a6b-b8d5-4695-b339-db0bdac8d17b","Type":"ContainerDied","Data":"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"}
Jul 30 21:28:58.415500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.415340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8" event={"ID":"339c6a6b-b8d5-4695-b339-db0bdac8d17b","Type":"ContainerDied","Data":"b595396663ceb0d8df1ca752adcf398e5ddd029ae8fdfd7d629f96849b9fa73e"}
Jul 30 21:28:58.415500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.415360    2583 scope.go:117] "RemoveContainer" containerID="3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"
Jul 30 21:28:58.416684 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.416655    2583 generic.go:356] "Generic (PLEG): container finished" podID="31f38dcf-630d-4159-9876-9daa4612d5b5" containerID="2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74" exitCode=0
Jul 30 21:28:58.416795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.416690    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"
Jul 30 21:28:58.416795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.416721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" event={"ID":"31f38dcf-630d-4159-9876-9daa4612d5b5","Type":"ContainerDied","Data":"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"}
Jul 30 21:28:58.416795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.416764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph" event={"ID":"31f38dcf-630d-4159-9876-9daa4612d5b5","Type":"ContainerDied","Data":"1917080a88bd34bc18393e4a184a3927871314b804c0ac0fb6dec4c284cb5802"}
Jul 30 21:28:58.418709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.418461    2583 generic.go:356] "Generic (PLEG): container finished" podID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" containerID="1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361" exitCode=0
Jul 30 21:28:58.418709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.418509    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"
Jul 30 21:28:58.418709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.418530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" event={"ID":"bcb4a170-bf04-4211-a41b-a3e9745c8c03","Type":"ContainerDied","Data":"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"}
Jul 30 21:28:58.418709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.418593    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm" event={"ID":"bcb4a170-bf04-4211-a41b-a3e9745c8c03","Type":"ContainerDied","Data":"e4750711e2818da5e1bfcf6c2f818b9dc6d5eb483b6b2aa7f62d3e2a9fbe3a95"}
Jul 30 21:28:58.426500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.426427    2583 scope.go:117] "RemoveContainer" containerID="3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"
Jul 30 21:28:58.428632 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:58.427846    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5\": container with ID starting with 3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5 not found: ID does not exist" containerID="3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"
Jul 30 21:28:58.428632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.427884    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5"} err="failed to get container status \"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5\": rpc error: code = NotFound desc = could not find container \"3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5\": container with ID starting with 3e73e5e810f1d5faa6445e415f405fac66b89490c648a098111e8a118778e7a5 not found: ID does not exist"
Jul 30 21:28:58.428632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.427907    2583 scope.go:117] "RemoveContainer" containerID="2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"
Jul 30 21:28:58.440986 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.440510    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:58.444246 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.444224    2583 scope.go:117] "RemoveContainer" containerID="2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"
Jul 30 21:28:58.445002 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:58.444975    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74\": container with ID starting with 2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74 not found: ID does not exist" containerID="2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"
Jul 30 21:28:58.445105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.445014    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74"} err="failed to get container status \"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74\": rpc error: code = NotFound desc = could not find container \"2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74\": container with ID starting with 2031d4fed8baa6d088b9e42faf6a94c20527dd29ee48ff15e79786eb0cb54f74 not found: ID does not exist"
Jul 30 21:28:58.445105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.445041    2583 scope.go:117] "RemoveContainer" containerID="1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"
Jul 30 21:28:58.447389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.447368    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b3059c6e-8589f4c6b-lz5q8"]
Jul 30 21:28:58.454125 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.454100    2583 scope.go:117] "RemoveContainer" containerID="1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"
Jul 30 21:28:58.454411 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:28:58.454391    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361\": container with ID starting with 1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361 not found: ID does not exist" containerID="1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"
Jul 30 21:28:58.454470 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.454420    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361"} err="failed to get container status \"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361\": rpc error: code = NotFound desc = could not find container \"1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361\": container with ID starting with 1dbe154b4099dfa578b56c852e9976b97eac776d9eed55344954f4173b6e8361 not found: ID does not exist"
Jul 30 21:28:58.457215 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.457193    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:58.461410 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.461391    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3059c6e-67c45cbcb8-zw8bm"]
Jul 30 21:28:58.470540 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.470518    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:58.474190 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.474171    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b3059c6e-f9dc565d4-f8lph"]
Jul 30 21:28:58.675689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.675558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:58.770599 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.770549    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:28:58.776275 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.776239    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31f38dcf-630d-4159-9876-9daa4612d5b5" path="/var/lib/kubelet/pods/31f38dcf-630d-4159-9876-9daa4612d5b5/volumes"
Jul 30 21:28:58.776621 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.776608    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" path="/var/lib/kubelet/pods/339c6a6b-b8d5-4695-b339-db0bdac8d17b/volumes"
Jul 30 21:28:58.776884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.776873    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf461db-dd88-4911-aa56-988015407552" path="/var/lib/kubelet/pods/acf461db-dd88-4911-aa56-988015407552/volumes"
Jul 30 21:28:58.777188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.777177    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" path="/var/lib/kubelet/pods/bcb4a170-bf04-4211-a41b-a3e9745c8c03/volumes"
Jul 30 21:28:58.871772 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:28:58.871725    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:00.009306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:00.009251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:10.701045 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:10.701002    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:14.383939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:14.383901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:16.744110 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:16.744070    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:16.853034 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:16.852998    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:16.946259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:16.946222    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:17.046607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:17.046481    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:17.563381 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:17.563346    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:43.557179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:43.557136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:49.534464 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:49.534420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:49.639976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:49.639938    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:29:49.735502 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:29:49.735464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:00.134644 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"]
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134908    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" containerName="maas-api"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134922    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" containerName="maas-api"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134937    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31f38dcf-630d-4159-9876-9daa4612d5b5" containerName="payload-pre-processing"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134942    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="31f38dcf-630d-4159-9876-9daa4612d5b5" containerName="payload-pre-processing"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134949    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" containerName="payload-processing"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134954    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" containerName="payload-processing"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134963    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf461db-dd88-4911-aa56-988015407552" containerName="revoke-keys"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.134968    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf461db-dd88-4911-aa56-988015407552" containerName="revoke-keys"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.135023    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="31f38dcf-630d-4159-9876-9daa4612d5b5" containerName="payload-pre-processing"
Jul 30 21:30:00.135028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.135032    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="339c6a6b-b8d5-4695-b339-db0bdac8d17b" containerName="payload-processing"
Jul 30 21:30:00.135341 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.135041    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf461db-dd88-4911-aa56-988015407552" containerName="revoke-keys"
Jul 30 21:30:00.135341 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.135048    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcb4a170-bf04-4211-a41b-a3e9745c8c03" containerName="maas-api"
Jul 30 21:30:00.137948 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.137931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"
Jul 30 21:30:00.139971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.139949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:30:00.148448 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.148415    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"]
Jul 30 21:30:00.156827 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.156803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"
Jul 30 21:30:00.289557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.289527    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"]
Jul 30 21:30:00.292155 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:30:00.292127    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8908ed36_f259_4334_8caa_abcb1c18db9f.slice/crio-eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310 WatchSource:0}: Error finding container eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310: Status 404 returned error can't find the container with id eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310
Jul 30 21:30:00.725404 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.725362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb" event={"ID":"8908ed36-f259-4334-8caa-abcb1c18db9f","Type":"ContainerStarted","Data":"020eac204f2f7cc777ae51b11f4fcfec0acd6175668c832cec9bac47e9a9f888"}
Jul 30 21:30:00.725642 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.725434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb" event={"ID":"8908ed36-f259-4334-8caa-abcb1c18db9f","Type":"ContainerStarted","Data":"eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310"}
Jul 30 21:30:00.741903 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:00.741852    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb" podStartSLOduration=0.741836033 podStartE2EDuration="741.836033ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:30:00.740536439 +0000 UTC m=+1972.506498318" watchObservedRunningTime="2026-07-30 21:30:00.741836033 +0000 UTC m=+1972.507797949"
Jul 30 21:30:01.731853 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:01.731816    2583 generic.go:356] "Generic (PLEG): container finished" podID="8908ed36-f259-4334-8caa-abcb1c18db9f" containerID="020eac204f2f7cc777ae51b11f4fcfec0acd6175668c832cec9bac47e9a9f888" exitCode=0
Jul 30 21:30:01.732280 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:01.731908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb" event={"ID":"8908ed36-f259-4334-8caa-abcb1c18db9f","Type":"ContainerDied","Data":"020eac204f2f7cc777ae51b11f4fcfec0acd6175668c832cec9bac47e9a9f888"}
Jul 30 21:30:02.866650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:02.866623    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"
Jul 30 21:30:03.743498 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:03.743459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb" event={"ID":"8908ed36-f259-4334-8caa-abcb1c18db9f","Type":"ContainerDied","Data":"eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310"}
Jul 30 21:30:03.743498 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:03.743499    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29757450-sptcb"
Jul 30 21:30:03.743731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:03.743511    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb0d7b10bce4236a0c8e1bc1761cf8c088260e330089a19d14c188ffcafd310"
Jul 30 21:30:05.675361 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:05.675319    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:11.325580 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:11.325534    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:11.465251 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:11.465221    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:12.278071 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:12.278029    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:22.201092 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:22.201053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:27.348761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:27.348725    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:33.505240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:33.505205    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:38.584222 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:38.584177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:39.352489 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:39.352445    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:39.452239 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:39.452180    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:39.554478 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:39.554428    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:39.653026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:39.652926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:39.752181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:39.752141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:40.709017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:40.708981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:40.808996 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:40.808958    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:42.445798 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:42.445760    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:48.306093 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:48.306038    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:49.707946 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:49.707901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:55.523303 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:55.523262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:55.625726 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:55.625687    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:55.866909 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:55.866816    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:56.274443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:56.274402    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:30:56.837510 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:30:56.837474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:00.508374 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:00.508333    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:00.599812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:00.599779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:01.800466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.800432    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:01.800886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.800756    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8908ed36-f259-4334-8caa-abcb1c18db9f" containerName="cleanup"
Jul 30 21:31:01.800886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.800791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8908ed36-f259-4334-8caa-abcb1c18db9f" containerName="cleanup"
Jul 30 21:31:01.800886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.800857    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8908ed36-f259-4334-8caa-abcb1c18db9f" containerName="cleanup"
Jul 30 21:31:01.803612 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.803596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:01.806065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.806043    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c76a3a24-serving-cert\""
Jul 30 21:31:01.806193 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.806080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:31:01.820536 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.820513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:01.959166 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.959128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:01.959342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.959200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkpn\" (UniqueName: \"kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:01.959342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:01.959232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.058368 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.058275    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:02.060337 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.060309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.060477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.060394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.060477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.060454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkpn\" (UniqueName: \"kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.060609 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:31:02.060532    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-serving-cert: secret "maas-api-e2e-mt-c76a3a24-serving-cert" not found
Jul 30 21:31:02.060674 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:31:02.060623    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls podName:378d6779-99bd-427f-bd91-176f332eeb21 nodeName:}" failed. No retries permitted until 2026-07-30 21:31:02.56060628 +0000 UTC m=+2034.326568158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls") pod "maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" (UID: "378d6779-99bd-427f-bd91-176f332eeb21") : secret "maas-api-e2e-mt-c76a3a24-serving-cert" not found
Jul 30 21:31:02.060949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.060929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.070067 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.070040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkpn\" (UniqueName: \"kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.359065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.358971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:02.429827 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.429781    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:02.430158 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.430139    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:02.433797 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.433777    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.435775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.435757    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c76a3a24\""
Jul 30 21:31:02.435896 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.435882    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c76a3a24-dockercfg-k6gmv\""
Jul 30 21:31:02.443619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.443592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:02.463410 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.463383    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:02.466903 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.466886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.475786 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.475764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:02.564684 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.564641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.564880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.564707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.564880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.564821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5r6\" (UniqueName: \"kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.564880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.564860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.565031 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.564883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck747\" (UniqueName: \"kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.567476 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.567454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") pod \"maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.666082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.665989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.666082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.666061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5r6\" (UniqueName: \"kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.666313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.666102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck747\" (UniqueName: \"kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.666313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.666155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.666968 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.666943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.667044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.666963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.673765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.673735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5r6\" (UniqueName: \"kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6\") pod \"payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") " pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.673883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.673786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck747\" (UniqueName: \"kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747\") pod \"payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.714746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.714716    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:02.745212 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.745178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:02.777843 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.777797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:02.884788 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.884695    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:02.887890 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:31:02.887748    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378d6779_99bd_427f_bd91_176f332eeb21.slice/crio-17bf3891c08fcdd55f37eac0b394333441146dac5d9afd851b1be5e9a7644efd WatchSource:0}: Error finding container 17bf3891c08fcdd55f37eac0b394333441146dac5d9afd851b1be5e9a7644efd: Status 404 returned error can't find the container with id 17bf3891c08fcdd55f37eac0b394333441146dac5d9afd851b1be5e9a7644efd
Jul 30 21:31:02.916741 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.916178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:02.919490 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:31:02.919459    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b3e354_6a05_40f8_b42e_569dbf2c42f6.slice/crio-df9a0964aac8dc6a52bef6646d6254e4f741c74a5305175928313e80e9ea227b WatchSource:0}: Error finding container df9a0964aac8dc6a52bef6646d6254e4f741c74a5305175928313e80e9ea227b: Status 404 returned error can't find the container with id df9a0964aac8dc6a52bef6646d6254e4f741c74a5305175928313e80e9ea227b
Jul 30 21:31:02.944330 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:02.944306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:02.952666 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:31:02.952639    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfbd692_64c4_4aa1_ad5d_ca428ff59e83.slice/crio-68c91bb86e1d7570bcd2e800793884ee305c7a02802a64a8516535c1a0d6b085 WatchSource:0}: Error finding container 68c91bb86e1d7570bcd2e800793884ee305c7a02802a64a8516535c1a0d6b085: Status 404 returned error can't find the container with id 68c91bb86e1d7570bcd2e800793884ee305c7a02802a64a8516535c1a0d6b085
Jul 30 21:31:03.024007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.023968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" event={"ID":"378d6779-99bd-427f-bd91-176f332eeb21","Type":"ContainerStarted","Data":"17bf3891c08fcdd55f37eac0b394333441146dac5d9afd851b1be5e9a7644efd"}
Jul 30 21:31:03.025848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.025806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" event={"ID":"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83","Type":"ContainerStarted","Data":"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"}
Jul 30 21:31:03.026143 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.025870    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" event={"ID":"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83","Type":"ContainerStarted","Data":"68c91bb86e1d7570bcd2e800793884ee305c7a02802a64a8516535c1a0d6b085"}
Jul 30 21:31:03.026143 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.025987    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:03.027404 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.027377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" event={"ID":"39b3e354-6a05-40f8-b42e-569dbf2c42f6","Type":"ContainerStarted","Data":"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"}
Jul 30 21:31:03.027507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.027425    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" event={"ID":"39b3e354-6a05-40f8-b42e-569dbf2c42f6","Type":"ContainerStarted","Data":"df9a0964aac8dc6a52bef6646d6254e4f741c74a5305175928313e80e9ea227b"}
Jul 30 21:31:03.027849 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.027832    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:03.041608 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.040826    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" podStartSLOduration=1.040808697 podStartE2EDuration="1.040808697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:31:03.040166913 +0000 UTC m=+2034.806128794" watchObservedRunningTime="2026-07-30 21:31:03.040808697 +0000 UTC m=+2034.806770576"
Jul 30 21:31:03.053652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:03.053608    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" podStartSLOduration=1.053592882 podStartE2EDuration="1.053592882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:31:03.053240957 +0000 UTC m=+2034.819202837" watchObservedRunningTime="2026-07-30 21:31:03.053592882 +0000 UTC m=+2034.819554761"
Jul 30 21:31:04.035212 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:04.035167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" event={"ID":"378d6779-99bd-427f-bd91-176f332eeb21","Type":"ContainerStarted","Data":"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"}
Jul 30 21:31:04.035735 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:04.035357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:04.049611 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:04.049543    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" podStartSLOduration=2.660526178 podStartE2EDuration="3.049526698s" totalImagesPullingTime="389.00052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:31:04.048789034 +0000 UTC m=+2035.814750913" watchObservedRunningTime="2026-07-30 21:31:04.049526698 +0000 UTC m=+2035.815488574"
Jul 30 21:31:10.046188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:10.046156    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:14.037695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:14.037662    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:14.038156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:14.037747    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:25.967864 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:25.967825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:27.198784 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:27.198747    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:27.294899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:27.294846    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:27.621926 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:27.621890    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:33.182338 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.182253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:33.241480 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.241443    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:33.681357 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.681317    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"]
Jul 30 21:31:33.684949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.684930    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:33.686787 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.686768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:31:33.692634 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.692609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"]
Jul 30 21:31:33.721152 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.721120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7\" (UID: \"546f2797-5efc-45df-81d5-4399f59992fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:33.822249 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.822214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7\" (UID: \"546f2797-5efc-45df-81d5-4399f59992fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:33.822864 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.822845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7\" (UID: \"546f2797-5efc-45df-81d5-4399f59992fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:33.996272 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:33.996237    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:34.138181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:34.138058    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"]
Jul 30 21:31:34.140994 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:31:34.140963    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546f2797_5efc_45df_81d5_4399f59992fd.slice/crio-7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b WatchSource:0}: Error finding container 7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b: Status 404 returned error can't find the container with id 7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b
Jul 30 21:31:34.189537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:34.189508    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7" event={"ID":"546f2797-5efc-45df-81d5-4399f59992fd","Type":"ContainerStarted","Data":"7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b"}
Jul 30 21:31:35.196902 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:35.196800    2583 generic.go:356] "Generic (PLEG): container finished" podID="546f2797-5efc-45df-81d5-4399f59992fd" containerID="d2af855f826843333de0b75fe2f2cd518d5a2375125494720d0a1c2bd687d195" exitCode=0
Jul 30 21:31:35.197279 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:35.196890    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7" event={"ID":"546f2797-5efc-45df-81d5-4399f59992fd","Type":"ContainerDied","Data":"d2af855f826843333de0b75fe2f2cd518d5a2375125494720d0a1c2bd687d195"}
Jul 30 21:31:36.338867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:36.338845    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:36.445119 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:36.445085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca\") pod \"546f2797-5efc-45df-81d5-4399f59992fd\" (UID: \"546f2797-5efc-45df-81d5-4399f59992fd\") "
Jul 30 21:31:36.445427 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:36.445402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "546f2797-5efc-45df-81d5-4399f59992fd" (UID: "546f2797-5efc-45df-81d5-4399f59992fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:31:36.546560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:36.546521    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546f2797-5efc-45df-81d5-4399f59992fd-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:37.207738 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:37.207703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7" event={"ID":"546f2797-5efc-45df-81d5-4399f59992fd","Type":"ContainerDied","Data":"7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b"}
Jul 30 21:31:37.207738 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:37.207737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"
Jul 30 21:31:37.207975 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:37.207787    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7907720a1e0700809a706804ff63a4863ba10c19cc5e7db1889579a5e844e65b"
Jul 30 21:31:44.252959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.252921    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:44.253431 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.253127    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" podUID="378d6779-99bd-427f-bd91-176f332eeb21" containerName="maas-api" containerID="cri-o://f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e" gracePeriod=30
Jul 30 21:31:44.281985 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.281952    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:44.316229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.316195    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:44.316549 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.316481    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" podUID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" containerName="payload-processing" containerID="cri-o://5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2" gracePeriod=30
Jul 30 21:31:44.324384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.324355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:44.324620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.324592    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" podUID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" containerName="payload-pre-processing" containerID="cri-o://6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd" gracePeriod=30
Jul 30 21:31:44.352208 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.352179    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:44.412997 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.412951    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:44.459701 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.459647    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"]
Jul 30 21:31:44.462623 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.462585    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c76a3a24-gm7w7"]
Jul 30 21:31:44.469732 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.469673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:44.598514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.598491    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:44.607607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.607586    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:44.618286 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.618260    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:44.710965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.710929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w5r6\" (UniqueName: \"kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6\") pod \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.710979    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") pod \"378d6779-99bd-427f-bd91-176f332eeb21\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711000    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvkpn\" (UniqueName: \"kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn\") pod \"378d6779-99bd-427f-bd91-176f332eeb21\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca\") pod \"378d6779-99bd-427f-bd91-176f332eeb21\" (UID: \"378d6779-99bd-427f-bd91-176f332eeb21\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume\") pod \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\" (UID: \"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck747\" (UniqueName: \"kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747\") pod \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") "
Jul 30 21:31:44.711167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume\") pod \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\" (UID: \"39b3e354-6a05-40f8-b42e-569dbf2c42f6\") "
Jul 30 21:31:44.711624 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711555    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "378d6779-99bd-427f-bd91-176f332eeb21" (UID: "378d6779-99bd-427f-bd91-176f332eeb21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:31:44.711749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711618    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" (UID: "1bfbd692-64c4-4aa1-ad5d-ca428ff59e83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:31:44.711749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.711662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39b3e354-6a05-40f8-b42e-569dbf2c42f6" (UID: "39b3e354-6a05-40f8-b42e-569dbf2c42f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:31:44.713397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.713372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6" (OuterVolumeSpecName: "kube-api-access-8w5r6") pod "1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" (UID: "1bfbd692-64c4-4aa1-ad5d-ca428ff59e83"). InnerVolumeSpecName "kube-api-access-8w5r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:31:44.713495 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.713404    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn" (OuterVolumeSpecName: "kube-api-access-fvkpn") pod "378d6779-99bd-427f-bd91-176f332eeb21" (UID: "378d6779-99bd-427f-bd91-176f332eeb21"). InnerVolumeSpecName "kube-api-access-fvkpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:31:44.713752 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.713734    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "378d6779-99bd-427f-bd91-176f332eeb21" (UID: "378d6779-99bd-427f-bd91-176f332eeb21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:31:44.713803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.713753    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747" (OuterVolumeSpecName: "kube-api-access-ck747") pod "39b3e354-6a05-40f8-b42e-569dbf2c42f6" (UID: "39b3e354-6a05-40f8-b42e-569dbf2c42f6"). InnerVolumeSpecName "kube-api-access-ck747". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:31:44.776644 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.776558    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="546f2797-5efc-45df-81d5-4399f59992fd" path="/var/lib/kubelet/pods/546f2797-5efc-45df-81d5-4399f59992fd/volumes"
Jul 30 21:31:44.812628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812603    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w5r6\" (UniqueName: \"kubernetes.io/projected/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-kube-api-access-8w5r6\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812629    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378d6779-99bd-427f-bd91-176f332eeb21-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812639    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvkpn\" (UniqueName: \"kubernetes.io/projected/378d6779-99bd-427f-bd91-176f332eeb21-kube-api-access-fvkpn\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812648    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378d6779-99bd-427f-bd91-176f332eeb21-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812658    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812667    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck747\" (UniqueName: \"kubernetes.io/projected/39b3e354-6a05-40f8-b42e-569dbf2c42f6-kube-api-access-ck747\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:44.812775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:44.812675    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39b3e354-6a05-40f8-b42e-569dbf2c42f6-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:31:45.006277 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.006240    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:45.248040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.248003    2583 generic.go:356] "Generic (PLEG): container finished" podID="378d6779-99bd-427f-bd91-176f332eeb21" containerID="f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e" exitCode=0
Jul 30 21:31:45.248231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.248062    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"
Jul 30 21:31:45.248231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.248096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" event={"ID":"378d6779-99bd-427f-bd91-176f332eeb21","Type":"ContainerDied","Data":"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"}
Jul 30 21:31:45.248231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.248161    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2" event={"ID":"378d6779-99bd-427f-bd91-176f332eeb21","Type":"ContainerDied","Data":"17bf3891c08fcdd55f37eac0b394333441146dac5d9afd851b1be5e9a7644efd"}
Jul 30 21:31:45.248231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.248189    2583 scope.go:117] "RemoveContainer" containerID="f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"
Jul 30 21:31:45.249946 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.249921    2583 generic.go:356] "Generic (PLEG): container finished" podID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" containerID="5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2" exitCode=0
Jul 30 21:31:45.250076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.249963    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"
Jul 30 21:31:45.250076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.250013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" event={"ID":"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83","Type":"ContainerDied","Data":"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"}
Jul 30 21:31:45.250076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.250060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg" event={"ID":"1bfbd692-64c4-4aa1-ad5d-ca428ff59e83","Type":"ContainerDied","Data":"68c91bb86e1d7570bcd2e800793884ee305c7a02802a64a8516535c1a0d6b085"}
Jul 30 21:31:45.251549 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.251529    2583 generic.go:356] "Generic (PLEG): container finished" podID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" containerID="6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd" exitCode=0
Jul 30 21:31:45.251677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.251612    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"
Jul 30 21:31:45.251677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.251621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" event={"ID":"39b3e354-6a05-40f8-b42e-569dbf2c42f6","Type":"ContainerDied","Data":"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"}
Jul 30 21:31:45.251677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.251652    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js" event={"ID":"39b3e354-6a05-40f8-b42e-569dbf2c42f6","Type":"ContainerDied","Data":"df9a0964aac8dc6a52bef6646d6254e4f741c74a5305175928313e80e9ea227b"}
Jul 30 21:31:45.259213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.259025    2583 scope.go:117] "RemoveContainer" containerID="f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"
Jul 30 21:31:45.259459 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:31:45.259322    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e\": container with ID starting with f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e not found: ID does not exist" containerID="f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"
Jul 30 21:31:45.259459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.259358    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e"} err="failed to get container status \"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e\": rpc error: code = NotFound desc = could not find container \"f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e\": container with ID starting with f388bf23a9a6b8eda38dd0ae16cb6adb2172026346e2f86905013351ed00424e not found: ID does not exist"
Jul 30 21:31:45.259459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.259375    2583 scope.go:117] "RemoveContainer" containerID="5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"
Jul 30 21:31:45.266281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.266255    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:45.267358 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.267318    2583 scope.go:117] "RemoveContainer" containerID="5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"
Jul 30 21:31:45.267720 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:31:45.267692    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2\": container with ID starting with 5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2 not found: ID does not exist" containerID="5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"
Jul 30 21:31:45.267807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.267730    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2"} err="failed to get container status \"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2\": rpc error: code = NotFound desc = could not find container \"5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2\": container with ID starting with 5eb236fee6a461ccb7717d23ed32c8856c5745d764b1b2ed19ded5d24adf21f2 not found: ID does not exist"
Jul 30 21:31:45.267807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.267752    2583 scope.go:117] "RemoveContainer" containerID="6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"
Jul 30 21:31:45.269300 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.269278    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c76a3a24-685fb58798-nt7r2"]
Jul 30 21:31:45.277684 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.277664    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:45.279775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.279757    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c76a3a24-76599959fd-5cdhg"]
Jul 30 21:31:45.282336 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.282322    2583 scope.go:117] "RemoveContainer" containerID="6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"
Jul 30 21:31:45.282600 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:31:45.282551    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd\": container with ID starting with 6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd not found: ID does not exist" containerID="6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"
Jul 30 21:31:45.282713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.282609    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd"} err="failed to get container status \"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd\": rpc error: code = NotFound desc = could not find container \"6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd\": container with ID starting with 6331b7739cbcfc0e11669b7b1e5e2d1e218d8cf52ecfe4d5d8143b50c33405fd not found: ID does not exist"
Jul 30 21:31:45.296361 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.296334    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:45.303827 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:45.303804    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c76a3a24-66966b84b8-lh5js"]
Jul 30 21:31:46.776997 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:46.776963    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" path="/var/lib/kubelet/pods/1bfbd692-64c4-4aa1-ad5d-ca428ff59e83/volumes"
Jul 30 21:31:46.777375 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:46.777319    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="378d6779-99bd-427f-bd91-176f332eeb21" path="/var/lib/kubelet/pods/378d6779-99bd-427f-bd91-176f332eeb21/volumes"
Jul 30 21:31:46.777630 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:46.777617    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" path="/var/lib/kubelet/pods/39b3e354-6a05-40f8-b42e-569dbf2c42f6/volumes"
Jul 30 21:31:50.720922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:50.720876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:50.778461 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:50.778424    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:51.129422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:51.129381    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:51.512927 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:51.512892    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:51.723463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:51.723421    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:51.822319 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:51.822170    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:52.023442 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:52.023402    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:57.843199 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:57.843154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:57.921092 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:57.921044    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:58.046387 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:58.046348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:58.145613 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:58.145489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:31:58.382365 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:31:58.382331    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:32:04.108638 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:04.108603    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:32:04.209074 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:04.209031    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:32:09.138601 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:09.138554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:32:09.141810 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:09.141782    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:32:34.664332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.664288    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:32:34.667550 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.664558    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" podUID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" containerName="manager" containerID="cri-o://75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3" gracePeriod=10
Jul 30 21:32:34.919207 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.919149    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:32:34.921629 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.921606    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume\") pod \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") "
Jul 30 21:32:34.921718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.921656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsc8l\" (UniqueName: \"kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l\") pod \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\" (UID: \"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada\") "
Jul 30 21:32:34.921982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.921959    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" (UID: "dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:32:34.923781 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:34.923757    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l" (OuterVolumeSpecName: "kube-api-access-xsc8l") pod "dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" (UID: "dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada"). InnerVolumeSpecName "kube-api-access-xsc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:32:35.022851 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.022817    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsc8l\" (UniqueName: \"kubernetes.io/projected/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-kube-api-access-xsc8l\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:32:35.022851 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.022843    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada-extensions-socket-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:32:35.495060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.495021    2583 generic.go:356] "Generic (PLEG): container finished" podID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" containerID="75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3" exitCode=0
Jul 30 21:32:35.495276 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.495070    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"
Jul 30 21:32:35.495276 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.495111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" event={"ID":"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada","Type":"ContainerDied","Data":"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"}
Jul 30 21:32:35.495276 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.495172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn" event={"ID":"dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada","Type":"ContainerDied","Data":"27c3df7d3dfbf36931d5ef8166a3afaeeb19cb49e315a4f9ab00082f351c1208"}
Jul 30 21:32:35.495276 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.495204    2583 scope.go:117] "RemoveContainer" containerID="75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"
Jul 30 21:32:35.505140 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.505123    2583 scope.go:117] "RemoveContainer" containerID="75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"
Jul 30 21:32:35.505393 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:32:35.505377    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3\": container with ID starting with 75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3 not found: ID does not exist" containerID="75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"
Jul 30 21:32:35.505438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.505402    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3"} err="failed to get container status \"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3\": rpc error: code = NotFound desc = could not find container \"75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3\": container with ID starting with 75d0a3396a0bab4fe37c5dd44c79b2c1f6986c1f2ebbb5505c782ba4d2aaf7e3 not found: ID does not exist"
Jul 30 21:32:35.515378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.515355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:32:35.521988 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:35.521967    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-htxtn"]
Jul 30 21:32:36.778278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:32:36.778240    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" path="/var/lib/kubelet/pods/dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada/volumes"
Jul 30 21:33:09.162636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:09.162539    2583 scope.go:117] "RemoveContainer" containerID="97d49e9c90fc91bf6a3eca81a48790e8a30ddaea1a2d4bd99a0e32ffdbb84ba7"
Jul 30 21:33:50.473070 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473034    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"]
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473366    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="378d6779-99bd-427f-bd91-176f332eeb21" containerName="maas-api"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473379    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="378d6779-99bd-427f-bd91-176f332eeb21" containerName="maas-api"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473391    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" containerName="payload-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473397    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" containerName="payload-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473404    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546f2797-5efc-45df-81d5-4399f59992fd" containerName="revoke-keys"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="546f2797-5efc-45df-81d5-4399f59992fd" containerName="revoke-keys"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473419    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" containerName="manager"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" containerName="manager"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473430    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" containerName="payload-pre-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473436    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" containerName="payload-pre-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473486    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bfbd692-64c4-4aa1-ad5d-ca428ff59e83" containerName="payload-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473495    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbe5e9ec-bf33-4abe-b7bc-4bf573b1fada" containerName="manager"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473501    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b3e354-6a05-40f8-b42e-569dbf2c42f6" containerName="payload-pre-processing"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473507    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="378d6779-99bd-427f-bd91-176f332eeb21" containerName="maas-api"
Jul 30 21:33:50.473605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.473511    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="546f2797-5efc-45df-81d5-4399f59992fd" containerName="revoke-keys"
Jul 30 21:33:50.476355 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.476338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.478155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.478133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6h2kb\""
Jul 30 21:33:50.486337 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.486312    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"]
Jul 30 21:33:50.522150 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.522120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b258b\" (UniqueName: \"kubernetes.io/projected/2267cd44-8396-4f33-868f-0e835620741f-kube-api-access-b258b\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.522297 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.522163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2267cd44-8396-4f33-868f-0e835620741f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.623121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.623089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b258b\" (UniqueName: \"kubernetes.io/projected/2267cd44-8396-4f33-868f-0e835620741f-kube-api-access-b258b\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.623282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.623130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2267cd44-8396-4f33-868f-0e835620741f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.623554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.623536    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2267cd44-8396-4f33-868f-0e835620741f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.638816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.638793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b258b\" (UniqueName: \"kubernetes.io/projected/2267cd44-8396-4f33-868f-0e835620741f-kube-api-access-b258b\") pod \"kuadrant-operator-controller-manager-68cf589589-8r25m\" (UID: \"2267cd44-8396-4f33-868f-0e835620741f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.786109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.786072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:50.928885 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:50.928833    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"]
Jul 30 21:33:50.933154 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:33:50.933129    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2267cd44_8396_4f33_868f_0e835620741f.slice/crio-1d2d43961fddd1bda82b90849b145c1801d5ddaf9f1af491cf49c96e01a6779c WatchSource:0}: Error finding container 1d2d43961fddd1bda82b90849b145c1801d5ddaf9f1af491cf49c96e01a6779c: Status 404 returned error can't find the container with id 1d2d43961fddd1bda82b90849b145c1801d5ddaf9f1af491cf49c96e01a6779c
Jul 30 21:33:51.859554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:51.859514    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m" event={"ID":"2267cd44-8396-4f33-868f-0e835620741f","Type":"ContainerStarted","Data":"763c68c7302b8218752bd2fcb33b00442678460787f85fcd7f0424aadc8b2bd9"}
Jul 30 21:33:51.860002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:51.859608    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:33:51.860002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:51.859635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m" event={"ID":"2267cd44-8396-4f33-868f-0e835620741f","Type":"ContainerStarted","Data":"1d2d43961fddd1bda82b90849b145c1801d5ddaf9f1af491cf49c96e01a6779c"}
Jul 30 21:33:51.876134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:33:51.876087    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m" podStartSLOduration=1.8760729980000002 podStartE2EDuration="1.876072998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:33:51.875018413 +0000 UTC m=+2203.640980291" watchObservedRunningTime="2026-07-30 21:33:51.876072998 +0000 UTC m=+2203.642034876"
Jul 30 21:34:02.866900 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:02.866864    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-8r25m"
Jul 30 21:34:09.184370 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:09.184334    2583 scope.go:117] "RemoveContainer" containerID="c46d88fbaadf7f0620be59f8a5c57d5bac2b8eb3d7742c838460d4b4598ca134"
Jul 30 21:34:09.626769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:09.626733    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:09.929263 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:09.929178    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:10.097214 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:10.097175    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:12.477457 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:12.477414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:12.570005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:12.569960    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:13.643500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.643467    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:34:13.647345 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.647323    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.649092 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.649065    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f08baaf8-serving-cert\""
Jul 30 21:34:13.649265 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.649179    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:34:13.656397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.656361    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:34:13.682117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.682066    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:13.706134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.706094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdhv\" (UniqueName: \"kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.706304 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.706148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.706304 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.706180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.806718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.806680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdhv\" (UniqueName: \"kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.806905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.806733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.806905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.806763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.806905 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:34:13.806866    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-serving-cert: secret "maas-api-e2e-mt-f08baaf8-serving-cert" not found
Jul 30 21:34:13.807039 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:34:13.806931    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls podName:0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d nodeName:}" failed. No retries permitted until 2026-07-30 21:34:14.306910636 +0000 UTC m=+2226.072872500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls") pod "maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" (UID: "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d") : secret "maas-api-e2e-mt-f08baaf8-serving-cert" not found
Jul 30 21:34:13.807532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.807506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:13.819993 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:13.819962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdhv\" (UniqueName: \"kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:14.181234 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.181193    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:14.280087 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.280045    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:14.305202 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.305163    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:34:14.308715 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.308686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.310411 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.310389    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f08baaf8\""
Jul 30 21:34:14.310695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.310673    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f08baaf8-dockercfg-swr6m\""
Jul 30 21:34:14.310849 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.310747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:14.313520 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.313493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") pod \"maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:14.316637 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.316610    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:34:14.365595 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.365544    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:34:14.368710 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.368692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.377434 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.377406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:34:14.411769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.411725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2f55\" (UniqueName: \"kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.411958 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.411784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbj9l\" (UniqueName: \"kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.411958 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.411861    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.412078 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.411953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.512706 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.512666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.512889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.512747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.512889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.512817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2f55\" (UniqueName: \"kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.512889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.512851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbj9l\" (UniqueName: \"kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.513426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.513407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.513717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.513692    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.525895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.525865    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2f55\" (UniqueName: \"kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55\") pod \"payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") " pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.526030 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.525910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbj9l\" (UniqueName: \"kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l\") pod \"payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.562417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.562379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:14.629308 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.629039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.680836 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.680550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.730012 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.729907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:34:14.734624 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.734598    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:34:14.809901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.809627    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:34:14.812927 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:34:14.812884    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95978af5_4a3f_4462_9f71_1b542de2c7e1.slice/crio-cc98f765d57958e19e8a98055654d5272bfe3f67448cce88cb2fb0fb6b2511d8 WatchSource:0}: Error finding container cc98f765d57958e19e8a98055654d5272bfe3f67448cce88cb2fb0fb6b2511d8: Status 404 returned error can't find the container with id cc98f765d57958e19e8a98055654d5272bfe3f67448cce88cb2fb0fb6b2511d8
Jul 30 21:34:14.870612 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.870588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:34:14.873099 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:34:14.873055    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518e9e57_d38c_4a97_a804_97b49baee793.slice/crio-b0c6c928d968a5d3aa3da9aaf1684898d7ab4c0280542650fb14b857df9486cf WatchSource:0}: Error finding container b0c6c928d968a5d3aa3da9aaf1684898d7ab4c0280542650fb14b857df9486cf: Status 404 returned error can't find the container with id b0c6c928d968a5d3aa3da9aaf1684898d7ab4c0280542650fb14b857df9486cf
Jul 30 21:34:14.976761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.976718    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" event={"ID":"518e9e57-d38c-4a97-a804-97b49baee793","Type":"ContainerStarted","Data":"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"}
Jul 30 21:34:14.976925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.976789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" event={"ID":"518e9e57-d38c-4a97-a804-97b49baee793","Type":"ContainerStarted","Data":"b0c6c928d968a5d3aa3da9aaf1684898d7ab4c0280542650fb14b857df9486cf"}
Jul 30 21:34:14.976925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.976823    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:14.978546 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.978520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" event={"ID":"95978af5-4a3f-4462-9f71-1b542de2c7e1","Type":"ContainerStarted","Data":"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"}
Jul 30 21:34:14.978695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.978594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" event={"ID":"95978af5-4a3f-4462-9f71-1b542de2c7e1","Type":"ContainerStarted","Data":"cc98f765d57958e19e8a98055654d5272bfe3f67448cce88cb2fb0fb6b2511d8"}
Jul 30 21:34:14.978695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.978625    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:14.979733 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.979705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" event={"ID":"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d","Type":"ContainerStarted","Data":"2e4a9a59c8a59d82941b1a087ee9b676f21408bd999981297867ed89e75da3b2"}
Jul 30 21:34:14.992047 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:14.991997    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" podStartSLOduration=0.991980294 podStartE2EDuration="991.980294ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:34:14.989762606 +0000 UTC m=+2226.755724510" watchObservedRunningTime="2026-07-30 21:34:14.991980294 +0000 UTC m=+2226.757942173"
Jul 30 21:34:15.006167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:15.006111    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" podStartSLOduration=1.00609276 podStartE2EDuration="1.00609276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:34:15.005383655 +0000 UTC m=+2226.771345534" watchObservedRunningTime="2026-07-30 21:34:15.00609276 +0000 UTC m=+2226.772054639"
Jul 30 21:34:15.989049 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:15.989009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" event={"ID":"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d","Type":"ContainerStarted","Data":"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"}
Jul 30 21:34:15.989049 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:15.989124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:16.005399 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:16.005350    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" podStartSLOduration=2.363565809 podStartE2EDuration="3.005333584s" totalImagesPullingTime="641.767775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:34:16.003436461 +0000 UTC m=+2227.769398341" watchObservedRunningTime="2026-07-30 21:34:16.005333584 +0000 UTC m=+2227.771295462"
Jul 30 21:34:22.002090 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:22.002055    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:34:25.991688 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:25.991655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:34:25.992192 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:25.991727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:34:37.571040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:37.570955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:49.846447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.846410    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:34:49.850122 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.850104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:49.851816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.851789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f08baaf8\"/\"kube-root-ca.crt\""
Jul 30 21:34:49.852252 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.852229    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f08baaf8\"/\"default-dockercfg-4pb67\""
Jul 30 21:34:49.852397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.852269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f08baaf8\"/\"e2e-lifecycle-model-f08baaf8-kserve-self-signed-certs\""
Jul 30 21:34:49.852397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.852272    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f08baaf8\"/\"openshift-service-ca.crt\""
Jul 30 21:34:49.858215 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.858194    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:34:49.875633 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:49.875554    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:34:50.020237 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020201    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.020453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.020453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.020453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.020453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.020453 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.020437    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp57x\" (UniqueName: \"kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.121704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.121704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.121704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp57x\" (UniqueName: \"kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.121704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.122036 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.122036 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.121771    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.122183 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.122160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.122239 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.122195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.122239 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.122209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.124007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.123984    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.124255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.124236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.128524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.128501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp57x\" (UniqueName: \"kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x\") pod \"e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") " pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.160422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.160397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:34:50.300788 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:50.300727    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:34:50.305638 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:34:50.305607    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa2f33b7_2501_4895_97cd_567be366479e.slice/crio-e427adbe07e3de705a3cf9d1ac70756303d9b5236c50d2c5c71467afa2007f3e WatchSource:0}: Error finding container e427adbe07e3de705a3cf9d1ac70756303d9b5236c50d2c5c71467afa2007f3e: Status 404 returned error can't find the container with id e427adbe07e3de705a3cf9d1ac70756303d9b5236c50d2c5c71467afa2007f3e
Jul 30 21:34:51.161957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:51.161923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" event={"ID":"aa2f33b7-2501-4895-97cd-567be366479e","Type":"ContainerStarted","Data":"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"}
Jul 30 21:34:51.162343 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:51.161964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" event={"ID":"aa2f33b7-2501-4895-97cd-567be366479e","Type":"ContainerStarted","Data":"e427adbe07e3de705a3cf9d1ac70756303d9b5236c50d2c5c71467afa2007f3e"}
Jul 30 21:34:51.176845 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:34:51.176800    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" podStartSLOduration=2.176786438 podStartE2EDuration="2.176786438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:34:51.175217448 +0000 UTC m=+2262.941179326" watchObservedRunningTime="2026-07-30 21:34:51.176786438 +0000 UTC m=+2262.942748316"
Jul 30 21:35:00.160650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:00.160609    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:00.160650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:00.160655    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:00.173202 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:00.173172    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:00.216717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:00.216687    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:01.364971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:01.364930    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:01.465516 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:01.465458    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:09.203248 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:09.203213    2583 scope.go:117] "RemoveContainer" containerID="dde6333283b43ad497a23f0a5952f9b22ac8d95924dfb6ecebea63acd33be36a"
Jul 30 21:35:09.214035 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:09.213956    2583 scope.go:117] "RemoveContainer" containerID="ce050eeef6b01c035d386ec72bc618f1344afe2bf57d7ecc0b498574f8a08c49"
Jul 30 21:35:17.366673 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.366636    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"]
Jul 30 21:35:17.369895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.369877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:17.371621 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.371594    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:35:17.375750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.375728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"]
Jul 30 21:35:17.451013 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.450977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5\" (UID: \"9a270d45-4633-4600-8418-25086f2e0510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:17.552004 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.551964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5\" (UID: \"9a270d45-4633-4600-8418-25086f2e0510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:17.552678 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.552658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5\" (UID: \"9a270d45-4633-4600-8418-25086f2e0510\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:17.682370 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.682286    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:17.814805 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:17.814773    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"]
Jul 30 21:35:17.817618 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:35:17.817583    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a270d45_4633_4600_8418_25086f2e0510.slice/crio-cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6 WatchSource:0}: Error finding container cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6: Status 404 returned error can't find the container with id cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6
Jul 30 21:35:18.292812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:18.292730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5" event={"ID":"9a270d45-4633-4600-8418-25086f2e0510","Type":"ContainerStarted","Data":"141d34c1d6e5daa48d47c53a77cb1fc4826c7769c73e78a0885812e9dba55b16"}
Jul 30 21:35:18.292812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:18.292809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5" event={"ID":"9a270d45-4633-4600-8418-25086f2e0510","Type":"ContainerStarted","Data":"cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6"}
Jul 30 21:35:18.304494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:18.304447    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5" podStartSLOduration=1.3044317859999999 podStartE2EDuration="1.304431786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:35:18.303330855 +0000 UTC m=+2290.069292734" watchObservedRunningTime="2026-07-30 21:35:18.304431786 +0000 UTC m=+2290.070393698"
Jul 30 21:35:19.298405 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:19.298369    2583 generic.go:356] "Generic (PLEG): container finished" podID="9a270d45-4633-4600-8418-25086f2e0510" containerID="141d34c1d6e5daa48d47c53a77cb1fc4826c7769c73e78a0885812e9dba55b16" exitCode=0
Jul 30 21:35:19.298812 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:19.298460    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5" event={"ID":"9a270d45-4633-4600-8418-25086f2e0510","Type":"ContainerDied","Data":"141d34c1d6e5daa48d47c53a77cb1fc4826c7769c73e78a0885812e9dba55b16"}
Jul 30 21:35:20.438728 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:20.438702    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:20.578928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:20.578815    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca\") pod \"9a270d45-4633-4600-8418-25086f2e0510\" (UID: \"9a270d45-4633-4600-8418-25086f2e0510\") "
Jul 30 21:35:20.579177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:20.579154    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a270d45-4633-4600-8418-25086f2e0510" (UID: "9a270d45-4633-4600-8418-25086f2e0510"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:35:20.679607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:20.679538    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a270d45-4633-4600-8418-25086f2e0510-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:21.309378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:21.309341    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"
Jul 30 21:35:21.309378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:21.309372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5" event={"ID":"9a270d45-4633-4600-8418-25086f2e0510","Type":"ContainerDied","Data":"cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6"}
Jul 30 21:35:21.309606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:21.309419    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd80a59d975833d144ac6e112f172ba8251d801ee3266b744fc94fa3c0e928c6"
Jul 30 21:35:32.828736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.828689    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:35:32.829261 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.829230    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" podUID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" containerName="maas-api" containerID="cri-o://4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683" gracePeriod=30
Jul 30 21:35:32.856827 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.856793    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:32.881901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.881872    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:35:32.882109 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.882085    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" podUID="95978af5-4a3f-4462-9f71-1b542de2c7e1" containerName="payload-processing" containerID="cri-o://a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193" gracePeriod=30
Jul 30 21:35:32.896496 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.895980    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:35:32.897136 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.896999    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" podUID="518e9e57-d38c-4a97-a804-97b49baee793" containerName="payload-pre-processing" containerID="cri-o://a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f" gracePeriod=30
Jul 30 21:35:32.924619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.924538    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:32.983337 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:32.983306    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:33.050435 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.050405    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"]
Jul 30 21:35:33.053828 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.053804    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f08baaf8-2zkx5"]
Jul 30 21:35:33.064862 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.064834    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:33.159719 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.159685    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:35:33.178103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.178079    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:35:33.198024 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.197999    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:35:33.283760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283731    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca\") pod \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") "
Jul 30 21:35:33.283945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume\") pod \"518e9e57-d38c-4a97-a804-97b49baee793\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") "
Jul 30 21:35:33.283945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283801    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2f55\" (UniqueName: \"kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55\") pod \"95978af5-4a3f-4462-9f71-1b542de2c7e1\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") "
Jul 30 21:35:33.283945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") pod \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") "
Jul 30 21:35:33.283945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbj9l\" (UniqueName: \"kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l\") pod \"518e9e57-d38c-4a97-a804-97b49baee793\" (UID: \"518e9e57-d38c-4a97-a804-97b49baee793\") "
Jul 30 21:35:33.284167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fdhv\" (UniqueName: \"kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv\") pod \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\" (UID: \"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d\") "
Jul 30 21:35:33.284167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.283991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume\") pod \"95978af5-4a3f-4462-9f71-1b542de2c7e1\" (UID: \"95978af5-4a3f-4462-9f71-1b542de2c7e1\") "
Jul 30 21:35:33.284167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.284123    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "518e9e57-d38c-4a97-a804-97b49baee793" (UID: "518e9e57-d38c-4a97-a804-97b49baee793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:35:33.284167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.284136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" (UID: "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:35:33.284395 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.284269    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.284395 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.284288    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/518e9e57-d38c-4a97-a804-97b49baee793-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.284514 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.284490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95978af5-4a3f-4462-9f71-1b542de2c7e1" (UID: "95978af5-4a3f-4462-9f71-1b542de2c7e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:35:33.286739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.286699    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l" (OuterVolumeSpecName: "kube-api-access-jbj9l") pod "518e9e57-d38c-4a97-a804-97b49baee793" (UID: "518e9e57-d38c-4a97-a804-97b49baee793"). InnerVolumeSpecName "kube-api-access-jbj9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:35:33.287048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.287026    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55" (OuterVolumeSpecName: "kube-api-access-d2f55") pod "95978af5-4a3f-4462-9f71-1b542de2c7e1" (UID: "95978af5-4a3f-4462-9f71-1b542de2c7e1"). InnerVolumeSpecName "kube-api-access-d2f55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:35:33.289081 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.289055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" (UID: "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:35:33.290175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.290152    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv" (OuterVolumeSpecName: "kube-api-access-2fdhv") pod "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" (UID: "0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d"). InnerVolumeSpecName "kube-api-access-2fdhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:35:33.365974 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.365889    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" containerID="4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683" exitCode=0
Jul 30 21:35:33.365974 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.365938    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"
Jul 30 21:35:33.365974 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.365949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" event={"ID":"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d","Type":"ContainerDied","Data":"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"}
Jul 30 21:35:33.366216 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.366006    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976" event={"ID":"0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d","Type":"ContainerDied","Data":"2e4a9a59c8a59d82941b1a087ee9b676f21408bd999981297867ed89e75da3b2"}
Jul 30 21:35:33.366216 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.366037    2583 scope.go:117] "RemoveContainer" containerID="4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"
Jul 30 21:35:33.367410 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.367389    2583 generic.go:356] "Generic (PLEG): container finished" podID="518e9e57-d38c-4a97-a804-97b49baee793" containerID="a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f" exitCode=0
Jul 30 21:35:33.367508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.367478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" event={"ID":"518e9e57-d38c-4a97-a804-97b49baee793","Type":"ContainerDied","Data":"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"}
Jul 30 21:35:33.367597 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.367510    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"
Jul 30 21:35:33.367597 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.367529    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6" event={"ID":"518e9e57-d38c-4a97-a804-97b49baee793","Type":"ContainerDied","Data":"b0c6c928d968a5d3aa3da9aaf1684898d7ab4c0280542650fb14b857df9486cf"}
Jul 30 21:35:33.368947 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.368926    2583 generic.go:356] "Generic (PLEG): container finished" podID="95978af5-4a3f-4462-9f71-1b542de2c7e1" containerID="a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193" exitCode=0
Jul 30 21:35:33.369050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.368965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" event={"ID":"95978af5-4a3f-4462-9f71-1b542de2c7e1","Type":"ContainerDied","Data":"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"}
Jul 30 21:35:33.369050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.369002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8" event={"ID":"95978af5-4a3f-4462-9f71-1b542de2c7e1","Type":"ContainerDied","Data":"cc98f765d57958e19e8a98055654d5272bfe3f67448cce88cb2fb0fb6b2511d8"}
Jul 30 21:35:33.369050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.368979    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"
Jul 30 21:35:33.375847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.375810    2583 scope.go:117] "RemoveContainer" containerID="4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"
Jul 30 21:35:33.376164 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:35:33.376137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683\": container with ID starting with 4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683 not found: ID does not exist" containerID="4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"
Jul 30 21:35:33.376273 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.376173    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683"} err="failed to get container status \"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683\": rpc error: code = NotFound desc = could not find container \"4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683\": container with ID starting with 4eac7de3d94a3e2ebfedc880c282e8637dbc463a4f3c644d2552044438dd7683 not found: ID does not exist"
Jul 30 21:35:33.376273 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.376196    2583 scope.go:117] "RemoveContainer" containerID="a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"
Jul 30 21:35:33.384141 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384123    2583 scope.go:117] "RemoveContainer" containerID="a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"
Jul 30 21:35:33.384501 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:35:33.384476    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f\": container with ID starting with a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f not found: ID does not exist" containerID="a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384509    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f"} err="failed to get container status \"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f\": rpc error: code = NotFound desc = could not find container \"a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f\": container with ID starting with a908373d1df1b61a623d8ca8ae59b13360a88f5786669dc43e74bc9ade45c10f not found: ID does not exist"
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384532    2583 scope.go:117] "RemoveContainer" containerID="a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384728    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbj9l\" (UniqueName: \"kubernetes.io/projected/518e9e57-d38c-4a97-a804-97b49baee793-kube-api-access-jbj9l\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384751    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fdhv\" (UniqueName: \"kubernetes.io/projected/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-kube-api-access-2fdhv\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384762    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95978af5-4a3f-4462-9f71-1b542de2c7e1-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384770    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2f55\" (UniqueName: \"kubernetes.io/projected/95978af5-4a3f-4462-9f71-1b542de2c7e1-kube-api-access-d2f55\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.384796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.384778    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:33.388157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.388137    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:35:33.391922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.391900    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f08baaf8-849dc47bcf-6s976"]
Jul 30 21:35:33.394344 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.394325    2583 scope.go:117] "RemoveContainer" containerID="a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"
Jul 30 21:35:33.394617 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:35:33.394593    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193\": container with ID starting with a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193 not found: ID does not exist" containerID="a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"
Jul 30 21:35:33.394751 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.394619    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193"} err="failed to get container status \"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193\": rpc error: code = NotFound desc = could not find container \"a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193\": container with ID starting with a765a52cf6205d72d34ff88f57e09c46e9f8d0c8977f0a7d9d4075aa66e20193 not found: ID does not exist"
Jul 30 21:35:33.400655 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.400635    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:35:33.406516 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.406493    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f08baaf8-7fb44fb6c8-vl9t8"]
Jul 30 21:35:33.414720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.414702    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:35:33.418803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:33.418784    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f08baaf8-7cbd9cb8d8-vwjs6"]
Jul 30 21:35:34.777107 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:34.777073    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" path="/var/lib/kubelet/pods/0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d/volumes"
Jul 30 21:35:34.777513 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:34.777415    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="518e9e57-d38c-4a97-a804-97b49baee793" path="/var/lib/kubelet/pods/518e9e57-d38c-4a97-a804-97b49baee793/volumes"
Jul 30 21:35:34.777733 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:34.777718    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95978af5-4a3f-4462-9f71-1b542de2c7e1" path="/var/lib/kubelet/pods/95978af5-4a3f-4462-9f71-1b542de2c7e1/volumes"
Jul 30 21:35:34.777989 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:34.777978    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a270d45-4633-4600-8418-25086f2e0510" path="/var/lib/kubelet/pods/9a270d45-4633-4600-8418-25086f2e0510/volumes"
Jul 30 21:35:36.362483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:36.362440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:41.447284 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:41.447250    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:35:41.545371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:41.545338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:35:56.467038 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.466975    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" podUID="aa2f33b7-2501-4895-97cd-567be366479e" containerName="main" containerID="cri-o://77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314" gracePeriod=45
Jul 30 21:35:56.715166 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.715136    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:56.789146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp57x\" (UniqueName: \"kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm\") pod \"aa2f33b7-2501-4895-97cd-567be366479e\" (UID: \"aa2f33b7-2501-4895-97cd-567be366479e\") "
Jul 30 21:35:56.789603 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:35:56.789662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789600    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:35:56.789662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789615    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home" (OuterVolumeSpecName: "home") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:35:56.789662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.789628    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:56.791549 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.791526    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm" (OuterVolumeSpecName: "dshm") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:35:56.791780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.791761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:35:56.791780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.791772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x" (OuterVolumeSpecName: "kube-api-access-zp57x") pod "aa2f33b7-2501-4895-97cd-567be366479e" (UID: "aa2f33b7-2501-4895-97cd-567be366479e"). InnerVolumeSpecName "kube-api-access-zp57x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:35:56.890587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.890541    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:56.890747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.890596    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:56.890747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.890610    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2f33b7-2501-4895-97cd-567be366479e-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:56.890747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.890625    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp57x\" (UniqueName: \"kubernetes.io/projected/aa2f33b7-2501-4895-97cd-567be366479e-kube-api-access-zp57x\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:56.890747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:56.890639    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2f33b7-2501-4895-97cd-567be366479e-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:35:57.485868 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.485830    2583 generic.go:356] "Generic (PLEG): container finished" podID="aa2f33b7-2501-4895-97cd-567be366479e" containerID="77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314" exitCode=0
Jul 30 21:35:57.485868 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.485874    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"
Jul 30 21:35:57.486389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.485868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" event={"ID":"aa2f33b7-2501-4895-97cd-567be366479e","Type":"ContainerDied","Data":"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"}
Jul 30 21:35:57.486389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.485920    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj" event={"ID":"aa2f33b7-2501-4895-97cd-567be366479e","Type":"ContainerDied","Data":"e427adbe07e3de705a3cf9d1ac70756303d9b5236c50d2c5c71467afa2007f3e"}
Jul 30 21:35:57.486389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.485940    2583 scope.go:117] "RemoveContainer" containerID="77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"
Jul 30 21:35:57.495325 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.495299    2583 scope.go:117] "RemoveContainer" containerID="77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"
Jul 30 21:35:57.495589 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:35:57.495548    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314\": container with ID starting with 77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314 not found: ID does not exist" containerID="77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"
Jul 30 21:35:57.495670 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.495596    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314"} err="failed to get container status \"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314\": rpc error: code = NotFound desc = could not find container \"77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314\": container with ID starting with 77a3861f516e59681fd4076f645c64474763c675d4e685724a71cac5b98c2314 not found: ID does not exist"
Jul 30 21:35:57.504946 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.504925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:35:57.509375 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:57.509355    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f08baaf8/e2e-lifecycle-model-f08baaf8-kserve-7b497988c-8pksj"]
Jul 30 21:35:58.776709 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:35:58.776677    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa2f33b7-2501-4895-97cd-567be366479e" path="/var/lib/kubelet/pods/aa2f33b7-2501-4895-97cd-567be366479e/volumes"
Jul 30 21:36:18.848804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:18.848767    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:20.888917 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:20.888878    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:24.068890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:24.068850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:24.201771 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:24.201732    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:24.391792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:24.391696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:24.625884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:24.625850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:35.907253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:35.907210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:36.006238 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:36.006206    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:36.107155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:36.107121    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:36.715811 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:36.715761    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:36.814991 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:36.814954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:36.915246 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:36.915186    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:38.219462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:38.219420    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:44.633388 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:44.633350    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:44.693930 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:44.693887    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:45.337130 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:45.337088    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:50.987875 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:50.987834    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:51.055444 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:51.055410    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:51.124997 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:51.124956    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:36:51.224724 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:36:51.224684    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:01.701888 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:01.701847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:01.928182 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:01.928147    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:07.329893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:07.329846    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:09.164264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:09.164231    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:37:09.169926 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:09.169900    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:37:22.837980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:22.837931    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:28.973849 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:28.973787    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:29.043931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:29.043873    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:29.330816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:29.330776    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:29.634813 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:29.634700    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:29.828398 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:29.828313    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:32.521582 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:32.521527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:32.642059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:32.642023    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:33.622062 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622030    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622350    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" containerName="maas-api"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622361    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" containerName="maas-api"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622372    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="518e9e57-d38c-4a97-a804-97b49baee793" containerName="payload-pre-processing"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622378    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="518e9e57-d38c-4a97-a804-97b49baee793" containerName="payload-pre-processing"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622384    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a270d45-4633-4600-8418-25086f2e0510" containerName="revoke-keys"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622389    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a270d45-4633-4600-8418-25086f2e0510" containerName="revoke-keys"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622399    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95978af5-4a3f-4462-9f71-1b542de2c7e1" containerName="payload-processing"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622404    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="95978af5-4a3f-4462-9f71-1b542de2c7e1" containerName="payload-processing"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622415    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2f33b7-2501-4895-97cd-567be366479e" containerName="main"
Jul 30 21:37:33.622430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622420    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2f33b7-2501-4895-97cd-567be366479e" containerName="main"
Jul 30 21:37:33.622774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622488    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9587fa-cb99-4fe9-b81c-c1afe9d9ee3d" containerName="maas-api"
Jul 30 21:37:33.622774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622498    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="518e9e57-d38c-4a97-a804-97b49baee793" containerName="payload-pre-processing"
Jul 30 21:37:33.622774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622504    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2f33b7-2501-4895-97cd-567be366479e" containerName="main"
Jul 30 21:37:33.622774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622510    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="95978af5-4a3f-4462-9f71-1b542de2c7e1" containerName="payload-processing"
Jul 30 21:37:33.622774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.622515    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a270d45-4633-4600-8418-25086f2e0510" containerName="revoke-keys"
Jul 30 21:37:33.625639 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.625617    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.627359 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.627337    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:37:33.627359 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.627358    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b2dab5-serving-cert\""
Jul 30 21:37:33.634901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.634878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:37:33.662144 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.662109    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:33.763243 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.763213    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:33.766223 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.766203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.766333 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.766245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctjr\" (UniqueName: \"kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.766333 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.766316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.866780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.866745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.866983 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.866796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.866983 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.866916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctjr\" (UniqueName: \"kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.867532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.867509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.869442 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.869421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.878005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.877940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctjr\" (UniqueName: \"kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr\") pod \"maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:33.938177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:33.938149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:34.081171 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.081137    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:37:34.084812 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:37:34.084779    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2724666b_f878_4711_9e49_1cae6f99093e.slice/crio-acedd80077bf2e632cc67cf727a70dbfa05792b7aadf64ae9875458fa07f74a1 WatchSource:0}: Error finding container acedd80077bf2e632cc67cf727a70dbfa05792b7aadf64ae9875458fa07f74a1: Status 404 returned error can't find the container with id acedd80077bf2e632cc67cf727a70dbfa05792b7aadf64ae9875458fa07f74a1
Jul 30 21:37:34.152095 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.152004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:34.253219 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.253186    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:37:34.255847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.255819    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:34.257746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.257726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.259685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.259663    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b2dab5\""
Jul 30 21:37:34.259787 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.259710    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b2dab5-dockercfg-9h7r8\""
Jul 30 21:37:34.268179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.268159    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:37:34.294661 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.294632    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:37:34.297954 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.297937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.307999 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.307976    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:37:34.371250 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.371221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.371450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.371275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9kp\" (UniqueName: \"kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.371450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.371310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rqz\" (UniqueName: \"kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.371450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.371358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.471931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.471858    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rqz\" (UniqueName: \"kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.471931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.471905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.472161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.471947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.472161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.471992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9kp\" (UniqueName: \"kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.472609 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.472561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.472746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.472726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.479835 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.479809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rqz\" (UniqueName: \"kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz\") pod \"payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.479932 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.479857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9kp\" (UniqueName: \"kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp\") pod \"payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.570943 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.570911    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.609040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.609007    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.741386 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.733295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:37:34.741825 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:37:34.741390    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e655149_7318_4be1_82f1_8ebecb8cc97d.slice/crio-58fc37bebfec483080ca73321a2770299424e6a9b02783abb47989acfacbefe0 WatchSource:0}: Error finding container 58fc37bebfec483080ca73321a2770299424e6a9b02783abb47989acfacbefe0: Status 404 returned error can't find the container with id 58fc37bebfec483080ca73321a2770299424e6a9b02783abb47989acfacbefe0
Jul 30 21:37:34.777312 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.777288    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:37:34.779739 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:37:34.779712    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515a28ad_7804_446f_8c9b_2039982b50d5.slice/crio-3b94f1c4b38d1172e982dd35fbd960347d7ebf8338ef190f3d6a3424ddb4f4f1 WatchSource:0}: Error finding container 3b94f1c4b38d1172e982dd35fbd960347d7ebf8338ef190f3d6a3424ddb4f4f1: Status 404 returned error can't find the container with id 3b94f1c4b38d1172e982dd35fbd960347d7ebf8338ef190f3d6a3424ddb4f4f1
Jul 30 21:37:34.951753 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.951711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" event={"ID":"515a28ad-7804-446f-8c9b-2039982b50d5","Type":"ContainerStarted","Data":"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"}
Jul 30 21:37:34.951961 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.951788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" event={"ID":"515a28ad-7804-446f-8c9b-2039982b50d5","Type":"ContainerStarted","Data":"3b94f1c4b38d1172e982dd35fbd960347d7ebf8338ef190f3d6a3424ddb4f4f1"}
Jul 30 21:37:34.951961 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.951882    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:34.953435 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.953410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" event={"ID":"9e655149-7318-4be1-82f1-8ebecb8cc97d","Type":"ContainerStarted","Data":"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"}
Jul 30 21:37:34.953552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.953456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" event={"ID":"9e655149-7318-4be1-82f1-8ebecb8cc97d","Type":"ContainerStarted","Data":"58fc37bebfec483080ca73321a2770299424e6a9b02783abb47989acfacbefe0"}
Jul 30 21:37:34.953552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.953480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:34.955134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.955112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" event={"ID":"2724666b-f878-4711-9e49-1cae6f99093e","Type":"ContainerStarted","Data":"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"}
Jul 30 21:37:34.955229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.955149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" event={"ID":"2724666b-f878-4711-9e49-1cae6f99093e","Type":"ContainerStarted","Data":"acedd80077bf2e632cc67cf727a70dbfa05792b7aadf64ae9875458fa07f74a1"}
Jul 30 21:37:34.955229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.955167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:34.967965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.967922    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" podStartSLOduration=0.967903866 podStartE2EDuration="967.903866ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:37:34.965376871 +0000 UTC m=+2426.731338751" watchObservedRunningTime="2026-07-30 21:37:34.967903866 +0000 UTC m=+2426.733865790"
Jul 30 21:37:34.980663 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.980624    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" podStartSLOduration=0.980610379 podStartE2EDuration="980.610379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:37:34.97917077 +0000 UTC m=+2426.745132650" watchObservedRunningTime="2026-07-30 21:37:34.980610379 +0000 UTC m=+2426.746572258"
Jul 30 21:37:34.995358 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:34.995270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" podStartSLOduration=1.601153238 podStartE2EDuration="1.995254256s" totalImagesPullingTime="394.101018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:37:34.992506725 +0000 UTC m=+2426.758468606" watchObservedRunningTime="2026-07-30 21:37:34.995254256 +0000 UTC m=+2426.761216208"
Jul 30 21:37:40.965126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:40.965096    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:37:45.646446 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:45.646406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:45.701117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:45.701073    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:45.961729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:45.961647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:37:45.961729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:45.961720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:37:57.565779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:57.565743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:57.883807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:57.883720    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:58.192043 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:58.191955    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:37:58.533156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:37:58.533115    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:00.687486 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:00.687450    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:00.786879 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:00.786838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:01.782634 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.782603    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:38:01.786207 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.786188    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:01.787885 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.787857    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e79f82-serving-cert\""
Jul 30 21:38:01.795444 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.795046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:38:01.826504 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.826466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:01.912768 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.912731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:01.912972 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.912786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqjs\" (UniqueName: \"kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:01.912972 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.912904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:01.926882 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:01.926853    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:02.014025 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.013994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.014226 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.014041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.014226 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.014167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqjs\" (UniqueName: \"kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.014838 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.014812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.016721 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.016693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.025183 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.025159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqjs\" (UniqueName: \"kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs\") pod \"maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.099128 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.099038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:02.241251 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.241227    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:38:02.243080 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:38:02.243057    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de8097a_702b_4ff0_8eb7_614d7799bb3a.slice/crio-74653f0ed7a031554d847a7a71cb3f8e825f8f39b049ba07045e66c076ac867c WatchSource:0}: Error finding container 74653f0ed7a031554d847a7a71cb3f8e825f8f39b049ba07045e66c076ac867c: Status 404 returned error can't find the container with id 74653f0ed7a031554d847a7a71cb3f8e825f8f39b049ba07045e66c076ac867c
Jul 30 21:38:02.323311 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.323271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:02.422507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.422412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:02.456527 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.456492    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:38:02.461009 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.460991    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.463126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.463109    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e79f82\""
Jul 30 21:38:02.463224 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.463114    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e79f82-dockercfg-ql2qq\""
Jul 30 21:38:02.470605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.470581    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:38:02.505484 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.505449    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:38:02.508850 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.508833    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.517862 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.517835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:38:02.619165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.619142    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.619271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.619198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.619271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.619233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.619384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.619324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcdwg\" (UniqueName: \"kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.720829 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.720732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.720829 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.720800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.721023 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.720826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.721023 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.720864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcdwg\" (UniqueName: \"kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.721525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.721499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.721654 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.721553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.727998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.727973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf\") pod \"payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.728089 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.728071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcdwg\" (UniqueName: \"kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg\") pod \"payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.772117 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.772084    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:02.820269 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.820235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:02.969717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:02.969689    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:38:02.972372 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:38:02.972281    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d14139_584e_42a3_939c_73f2c8949ea6.slice/crio-46d68b921477ff64ae2b348850f30f1d4c15c76bdec0585ddea34315d50a5457 WatchSource:0}: Error finding container 46d68b921477ff64ae2b348850f30f1d4c15c76bdec0585ddea34315d50a5457: Status 404 returned error can't find the container with id 46d68b921477ff64ae2b348850f30f1d4c15c76bdec0585ddea34315d50a5457
Jul 30 21:38:03.091483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.091445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" event={"ID":"b0d14139-584e-42a3-939c-73f2c8949ea6","Type":"ContainerStarted","Data":"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"}
Jul 30 21:38:03.091770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.091751    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:03.091905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.091881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" event={"ID":"b0d14139-584e-42a3-939c-73f2c8949ea6","Type":"ContainerStarted","Data":"46d68b921477ff64ae2b348850f30f1d4c15c76bdec0585ddea34315d50a5457"}
Jul 30 21:38:03.093439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.093416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" event={"ID":"3de8097a-702b-4ff0-8eb7-614d7799bb3a","Type":"ContainerStarted","Data":"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"}
Jul 30 21:38:03.093650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.093623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" event={"ID":"3de8097a-702b-4ff0-8eb7-614d7799bb3a","Type":"ContainerStarted","Data":"74653f0ed7a031554d847a7a71cb3f8e825f8f39b049ba07045e66c076ac867c"}
Jul 30 21:38:03.093959 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.093944    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:03.126608 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.126541    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" podStartSLOduration=1.8053168689999999 podStartE2EDuration="2.126526072s" totalImagesPullingTime="321.209203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:38:03.125358921 +0000 UTC m=+2454.891320801" watchObservedRunningTime="2026-07-30 21:38:03.126526072 +0000 UTC m=+2454.892487951"
Jul 30 21:38:03.126761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.126641    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" podStartSLOduration=1.126637056 podStartE2EDuration="1.126637056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:38:03.11089594 +0000 UTC m=+2454.876857821" watchObservedRunningTime="2026-07-30 21:38:03.126637056 +0000 UTC m=+2454.892598935"
Jul 30 21:38:03.132333 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:03.132000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:38:03.135272 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:38:03.135245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3531cb9d_f507_406e_9fb6_9129731bba7d.slice/crio-8b4a615694caaf25ca0a3b9ce3ec90ed38a4718347040d1f6f916986d03b4b2c WatchSource:0}: Error finding container 8b4a615694caaf25ca0a3b9ce3ec90ed38a4718347040d1f6f916986d03b4b2c: Status 404 returned error can't find the container with id 8b4a615694caaf25ca0a3b9ce3ec90ed38a4718347040d1f6f916986d03b4b2c
Jul 30 21:38:04.101367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:04.101330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" event={"ID":"3531cb9d-f507-406e-9fb6-9129731bba7d","Type":"ContainerStarted","Data":"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"}
Jul 30 21:38:04.101957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:04.101934    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:04.102066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:04.101971    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" event={"ID":"3531cb9d-f507-406e-9fb6-9129731bba7d","Type":"ContainerStarted","Data":"8b4a615694caaf25ca0a3b9ce3ec90ed38a4718347040d1f6f916986d03b4b2c"}
Jul 30 21:38:04.117606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:04.116589    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" podStartSLOduration=2.116542967 podStartE2EDuration="2.116542967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:38:04.116103213 +0000 UTC m=+2455.882065093" watchObservedRunningTime="2026-07-30 21:38:04.116542967 +0000 UTC m=+2455.882504847"
Jul 30 21:38:09.107330 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:09.107298    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:38:09.264454 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:09.264421    2583 scope.go:117] "RemoveContainer" containerID="d2af855f826843333de0b75fe2f2cd518d5a2375125494720d0a1c2bd687d195"
Jul 30 21:38:14.104217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:14.104187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:38:15.108805 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:15.108770    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:38:16.223031 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:16.222990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:25.701310 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:25.701272    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:33.347081 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.347042    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:38:33.351096 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.351076    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.353082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.353060    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b2dab5\"/\"kube-root-ca.crt\""
Jul 30 21:38:33.353503 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.353484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b2dab5\"/\"openshift-service-ca.crt\""
Jul 30 21:38:33.353647 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.353502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b2dab5\"/\"test-model-b2dab5-kserve-self-signed-certs\""
Jul 30 21:38:33.353647 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.353487    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b2dab5\"/\"default-dockercfg-jwk4x\""
Jul 30 21:38:33.357613 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.357589    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:38:33.370235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.370210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:38:33.484704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.484895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484718    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.484895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.484895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.484895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484862    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.485058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.484917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxvf9\" (UniqueName: \"kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.585834 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.585804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.585843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.585975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxvf9\" (UniqueName: \"kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586309 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586309 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.586413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.586313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.588480 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.588455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.588711 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.588695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.593470 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.593447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxvf9\" (UniqueName: \"kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9\") pod \"test-model-b2dab5-kserve-5cfd4b8b45-8zd7p\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") " pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.661687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.661597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:33.801044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:33.801019    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:38:33.803525 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:38:33.803494    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063a0f04_7470_4c0f_9d27_b51642a7714e.slice/crio-c5a82ceec60340a7f2dc02f45b63300686da89e12692821055283bd13c9a2532 WatchSource:0}: Error finding container c5a82ceec60340a7f2dc02f45b63300686da89e12692821055283bd13c9a2532: Status 404 returned error can't find the container with id c5a82ceec60340a7f2dc02f45b63300686da89e12692821055283bd13c9a2532
Jul 30 21:38:34.255230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:34.255181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" event={"ID":"063a0f04-7470-4c0f-9d27-b51642a7714e","Type":"ContainerStarted","Data":"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"}
Jul 30 21:38:34.255230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:34.255238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" event={"ID":"063a0f04-7470-4c0f-9d27-b51642a7714e","Type":"ContainerStarted","Data":"c5a82ceec60340a7f2dc02f45b63300686da89e12692821055283bd13c9a2532"}
Jul 30 21:38:34.269780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:34.269729    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" podStartSLOduration=1.269715506 podStartE2EDuration="1.269715506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:38:34.268481393 +0000 UTC m=+2486.034443272" watchObservedRunningTime="2026-07-30 21:38:34.269715506 +0000 UTC m=+2486.035677385"
Jul 30 21:38:43.662641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:43.662599    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:43.663037 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:43.662659    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:43.675247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:43.675223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:38:44.316247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:38:44.316219    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:39:12.448533 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.448434    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:39:12.452706 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.452684    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.454977 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.454950    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e79f82\"/\"kube-root-ca.crt\""
Jul 30 21:39:12.454977 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.454974    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e79f82\"/\"openshift-service-ca.crt\""
Jul 30 21:39:12.455145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.455030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e79f82\"/\"default-dockercfg-x8pzc\""
Jul 30 21:39:12.455145 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.454959    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e79f82\"/\"test-model-e79f82-kserve-self-signed-certs\""
Jul 30 21:39:12.459965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.459942    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:39:12.528682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.528856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.528856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.528856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.529014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.529014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.528931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjm29\" (UniqueName: \"kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.540428 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.540391    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:39:12.630266 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630266 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjm29\" (UniqueName: \"kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.630884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.631001 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.630907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.631086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.631054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.633274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.633253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.633551 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.633532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.638032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.638011    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjm29\" (UniqueName: \"kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29\") pod \"test-model-e79f82-kserve-7db78667d7-tc6z2\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") " pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.764104 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.764070    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:12.901155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:12.901124    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:39:12.905066 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:39:12.905034    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd82df4e7_e480_4c9f_99f2_9bf7eacf83df.slice/crio-540167c987fc1686cd2f48c840dcc83d4d0492d3e04954f698fcfa1eef0453cf WatchSource:0}: Error finding container 540167c987fc1686cd2f48c840dcc83d4d0492d3e04954f698fcfa1eef0453cf: Status 404 returned error can't find the container with id 540167c987fc1686cd2f48c840dcc83d4d0492d3e04954f698fcfa1eef0453cf
Jul 30 21:39:13.448257 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:13.448220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" event={"ID":"d82df4e7-e480-4c9f-99f2-9bf7eacf83df","Type":"ContainerStarted","Data":"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"}
Jul 30 21:39:13.448257 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:13.448262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" event={"ID":"d82df4e7-e480-4c9f-99f2-9bf7eacf83df","Type":"ContainerStarted","Data":"540167c987fc1686cd2f48c840dcc83d4d0492d3e04954f698fcfa1eef0453cf"}
Jul 30 21:39:13.463975 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:13.463916    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" podStartSLOduration=1.463899428 podStartE2EDuration="1.463899428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:39:13.462092016 +0000 UTC m=+2525.228053895" watchObservedRunningTime="2026-07-30 21:39:13.463899428 +0000 UTC m=+2525.229861308"
Jul 30 21:39:22.764291 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:22.764246    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:22.764837 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:22.764314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:22.777391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:22.777357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:39:23.513447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:39:23.513416    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:40:01.084014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:01.083976    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:40:01.103554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:01.103523    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:02.022227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:02.022194    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:40:02.040744 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:02.040706    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:12.649245 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.649214    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:40:12.649698 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.649431    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" podUID="2724666b-f878-4711-9e49-1cae6f99093e" containerName="maas-api" containerID="cri-o://6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f" gracePeriod=30
Jul 30 21:40:12.674861 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.674829    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:12.706004 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.705958    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:40:12.706227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.706201    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" podUID="515a28ad-7804-446f-8c9b-2039982b50d5" containerName="payload-processing" containerID="cri-o://23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15" gracePeriod=30
Jul 30 21:40:12.713548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.712971    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:40:12.750949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.750901    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" podUID="9e655149-7318-4be1-82f1-8ebecb8cc97d" containerName="payload-pre-processing" containerID="cri-o://a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b" gracePeriod=30
Jul 30 21:40:12.778543 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.778512    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:12.869275 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.869234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:12.928436 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:12.928398    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:13.019677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.019617    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:40:13.046808 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.046782    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:40:13.116231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.116206    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:40:13.179713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.179621    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume\") pod \"515a28ad-7804-446f-8c9b-2039982b50d5\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") "
Jul 30 21:40:13.179713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.179661    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85rqz\" (UniqueName: \"kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz\") pod \"9e655149-7318-4be1-82f1-8ebecb8cc97d\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") "
Jul 30 21:40:13.179713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.179696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj9kp\" (UniqueName: \"kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp\") pod \"515a28ad-7804-446f-8c9b-2039982b50d5\" (UID: \"515a28ad-7804-446f-8c9b-2039982b50d5\") "
Jul 30 21:40:13.180005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.179783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume\") pod \"9e655149-7318-4be1-82f1-8ebecb8cc97d\" (UID: \"9e655149-7318-4be1-82f1-8ebecb8cc97d\") "
Jul 30 21:40:13.180083 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.180061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "515a28ad-7804-446f-8c9b-2039982b50d5" (UID: "515a28ad-7804-446f-8c9b-2039982b50d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:13.180196 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.180167    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e655149-7318-4be1-82f1-8ebecb8cc97d" (UID: "9e655149-7318-4be1-82f1-8ebecb8cc97d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:13.181998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.181978    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp" (OuterVolumeSpecName: "kube-api-access-pj9kp") pod "515a28ad-7804-446f-8c9b-2039982b50d5" (UID: "515a28ad-7804-446f-8c9b-2039982b50d5"). InnerVolumeSpecName "kube-api-access-pj9kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:13.182087 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.182003    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz" (OuterVolumeSpecName: "kube-api-access-85rqz") pod "9e655149-7318-4be1-82f1-8ebecb8cc97d" (UID: "9e655149-7318-4be1-82f1-8ebecb8cc97d"). InnerVolumeSpecName "kube-api-access-85rqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:13.280509 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cctjr\" (UniqueName: \"kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr\") pod \"2724666b-f878-4711-9e49-1cae6f99093e\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") "
Jul 30 21:40:13.280737 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls\") pod \"2724666b-f878-4711-9e49-1cae6f99093e\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") "
Jul 30 21:40:13.280737 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca\") pod \"2724666b-f878-4711-9e49-1cae6f99093e\" (UID: \"2724666b-f878-4711-9e49-1cae6f99093e\") "
Jul 30 21:40:13.280869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280799    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515a28ad-7804-446f-8c9b-2039982b50d5-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.280869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280812    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rqz\" (UniqueName: \"kubernetes.io/projected/9e655149-7318-4be1-82f1-8ebecb8cc97d-kube-api-access-85rqz\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.280869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280821    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9kp\" (UniqueName: \"kubernetes.io/projected/515a28ad-7804-446f-8c9b-2039982b50d5-kube-api-access-pj9kp\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.280869 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280829    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e655149-7318-4be1-82f1-8ebecb8cc97d-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.281027 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.280903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2724666b-f878-4711-9e49-1cae6f99093e" (UID: "2724666b-f878-4711-9e49-1cae6f99093e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:13.282731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.282707    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2724666b-f878-4711-9e49-1cae6f99093e" (UID: "2724666b-f878-4711-9e49-1cae6f99093e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:40:13.282810 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.282775    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr" (OuterVolumeSpecName: "kube-api-access-cctjr") pod "2724666b-f878-4711-9e49-1cae6f99093e" (UID: "2724666b-f878-4711-9e49-1cae6f99093e"). InnerVolumeSpecName "kube-api-access-cctjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:13.289068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.289038    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:13.381508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.381473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cctjr\" (UniqueName: \"kubernetes.io/projected/2724666b-f878-4711-9e49-1cae6f99093e-kube-api-access-cctjr\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.381508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.381503    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2724666b-f878-4711-9e49-1cae6f99093e-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.381508 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.381514    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2724666b-f878-4711-9e49-1cae6f99093e-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:13.757128 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.757090    2583 generic.go:356] "Generic (PLEG): container finished" podID="515a28ad-7804-446f-8c9b-2039982b50d5" containerID="23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15" exitCode=0
Jul 30 21:40:13.757625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.757155    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"
Jul 30 21:40:13.757625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.757176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" event={"ID":"515a28ad-7804-446f-8c9b-2039982b50d5","Type":"ContainerDied","Data":"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"}
Jul 30 21:40:13.757625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.757262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4" event={"ID":"515a28ad-7804-446f-8c9b-2039982b50d5","Type":"ContainerDied","Data":"3b94f1c4b38d1172e982dd35fbd960347d7ebf8338ef190f3d6a3424ddb4f4f1"}
Jul 30 21:40:13.757625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.757295    2583 scope.go:117] "RemoveContainer" containerID="23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"
Jul 30 21:40:13.758704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.758469    2583 generic.go:356] "Generic (PLEG): container finished" podID="9e655149-7318-4be1-82f1-8ebecb8cc97d" containerID="a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b" exitCode=0
Jul 30 21:40:13.758704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.758513    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"
Jul 30 21:40:13.758704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.758545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" event={"ID":"9e655149-7318-4be1-82f1-8ebecb8cc97d","Type":"ContainerDied","Data":"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"}
Jul 30 21:40:13.758704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.758626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82" event={"ID":"9e655149-7318-4be1-82f1-8ebecb8cc97d","Type":"ContainerDied","Data":"58fc37bebfec483080ca73321a2770299424e6a9b02783abb47989acfacbefe0"}
Jul 30 21:40:13.759977 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.759953    2583 generic.go:356] "Generic (PLEG): container finished" podID="2724666b-f878-4711-9e49-1cae6f99093e" containerID="6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f" exitCode=0
Jul 30 21:40:13.760101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.760060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" event={"ID":"2724666b-f878-4711-9e49-1cae6f99093e","Type":"ContainerDied","Data":"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"}
Jul 30 21:40:13.760167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.760098    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl" event={"ID":"2724666b-f878-4711-9e49-1cae6f99093e","Type":"ContainerDied","Data":"acedd80077bf2e632cc67cf727a70dbfa05792b7aadf64ae9875458fa07f74a1"}
Jul 30 21:40:13.760167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.760129    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"
Jul 30 21:40:13.768868 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.768853    2583 scope.go:117] "RemoveContainer" containerID="23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"
Jul 30 21:40:13.769143 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:13.769126    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15\": container with ID starting with 23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15 not found: ID does not exist" containerID="23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"
Jul 30 21:40:13.769205 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.769152    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15"} err="failed to get container status \"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15\": rpc error: code = NotFound desc = could not find container \"23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15\": container with ID starting with 23caa42e9bce3f7ac9fc0de26504b0913312725e0f62f1ca7307d12a4da39b15 not found: ID does not exist"
Jul 30 21:40:13.769205 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.769168    2583 scope.go:117] "RemoveContainer" containerID="a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"
Jul 30 21:40:13.777907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.777887    2583 scope.go:117] "RemoveContainer" containerID="a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"
Jul 30 21:40:13.778160 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:13.778140    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b\": container with ID starting with a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b not found: ID does not exist" containerID="a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"
Jul 30 21:40:13.778211 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.778167    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b"} err="failed to get container status \"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b\": rpc error: code = NotFound desc = could not find container \"a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b\": container with ID starting with a17ef6501753d2ef456eb788f46d71e8025f1f627472ffe8083cd45bd69ff29b not found: ID does not exist"
Jul 30 21:40:13.778211 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.778183    2583 scope.go:117] "RemoveContainer" containerID="6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"
Jul 30 21:40:13.779716 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.779698    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:40:13.786183 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.786158    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b2dab5-65b86bd746-btvv4"]
Jul 30 21:40:13.787795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.787772    2583 scope.go:117] "RemoveContainer" containerID="6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"
Jul 30 21:40:13.788085 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:13.788064    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f\": container with ID starting with 6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f not found: ID does not exist" containerID="6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"
Jul 30 21:40:13.788160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.788089    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f"} err="failed to get container status \"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f\": rpc error: code = NotFound desc = could not find container \"6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f\": container with ID starting with 6783c2e830b8e631c4f3f92cda2d7c6dd5e3c498f1b1ae83fcf3954c1a1db40f not found: ID does not exist"
Jul 30 21:40:13.794762 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.794714    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:40:13.796560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.796539    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b2dab5-549d4464dd-zfr82"]
Jul 30 21:40:13.805169 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.805147    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:40:13.808958 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:13.808938    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dab5-5448d8bcc9-vtkwl"]
Jul 30 21:40:14.776641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:14.776609    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2724666b-f878-4711-9e49-1cae6f99093e" path="/var/lib/kubelet/pods/2724666b-f878-4711-9e49-1cae6f99093e/volumes"
Jul 30 21:40:14.777013 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:14.776950    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515a28ad-7804-446f-8c9b-2039982b50d5" path="/var/lib/kubelet/pods/515a28ad-7804-446f-8c9b-2039982b50d5/volumes"
Jul 30 21:40:14.777243 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:14.777230    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e655149-7318-4be1-82f1-8ebecb8cc97d" path="/var/lib/kubelet/pods/9e655149-7318-4be1-82f1-8ebecb8cc97d/volumes"
Jul 30 21:40:16.105198 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.105153    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" podUID="063a0f04-7470-4c0f-9d27-b51642a7714e" containerName="main" containerID="cri-o://79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7" gracePeriod=45
Jul 30 21:40:16.359916 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.359858    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:40:16.509157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509167    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxvf9\" (UniqueName: \"kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509215    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509240    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509419    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm\") pod \"063a0f04-7470-4c0f-9d27-b51642a7714e\" (UID: \"063a0f04-7470-4c0f-9d27-b51642a7714e\") "
Jul 30 21:40:16.509532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509428    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home" (OuterVolumeSpecName: "home") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:16.509532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509493    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache" (OuterVolumeSpecName: "model-cache") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:16.509532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509510    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:16.509748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509695    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.509748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509709    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.509748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.509736    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.511676 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.511644    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:40:16.511783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.511701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9" (OuterVolumeSpecName: "kube-api-access-cxvf9") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "kube-api-access-cxvf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:16.511783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.511709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm" (OuterVolumeSpecName: "dshm") pod "063a0f04-7470-4c0f-9d27-b51642a7714e" (UID: "063a0f04-7470-4c0f-9d27-b51642a7714e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:16.610505 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.610418    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxvf9\" (UniqueName: \"kubernetes.io/projected/063a0f04-7470-4c0f-9d27-b51642a7714e-kube-api-access-cxvf9\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.610505 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.610449    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063a0f04-7470-4c0f-9d27-b51642a7714e-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.610505 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.610458    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063a0f04-7470-4c0f-9d27-b51642a7714e-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:16.783176 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.783137    2583 generic.go:356] "Generic (PLEG): container finished" podID="063a0f04-7470-4c0f-9d27-b51642a7714e" containerID="79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7" exitCode=0
Jul 30 21:40:16.783351 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.783253    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"
Jul 30 21:40:16.784011 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.783985    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" event={"ID":"063a0f04-7470-4c0f-9d27-b51642a7714e","Type":"ContainerDied","Data":"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"}
Jul 30 21:40:16.784162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.784025    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p" event={"ID":"063a0f04-7470-4c0f-9d27-b51642a7714e","Type":"ContainerDied","Data":"c5a82ceec60340a7f2dc02f45b63300686da89e12692821055283bd13c9a2532"}
Jul 30 21:40:16.784162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.784049    2583 scope.go:117] "RemoveContainer" containerID="79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"
Jul 30 21:40:16.797670 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.797649    2583 scope.go:117] "RemoveContainer" containerID="79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"
Jul 30 21:40:16.797938 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:16.797919    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7\": container with ID starting with 79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7 not found: ID does not exist" containerID="79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"
Jul 30 21:40:16.797984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.797948    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7"} err="failed to get container status \"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7\": rpc error: code = NotFound desc = could not find container \"79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7\": container with ID starting with 79b946f8ccdc8a8bfcc6aaff04db90a058f20b62e7e51cb0cf75827412828cc7 not found: ID does not exist"
Jul 30 21:40:16.808537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.808511    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:40:16.811872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:16.811850    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b2dab5/test-model-b2dab5-kserve-5cfd4b8b45-8zd7p"]
Jul 30 21:40:17.042687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.042641    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" podUID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" containerName="main" containerID="cri-o://d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25" gracePeriod=45
Jul 30 21:40:17.288980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.288955    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:40:17.418872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.418872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.418872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418873    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.419114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418897    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjm29\" (UniqueName: \"kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.419114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.419114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.418968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir\") pod \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\" (UID: \"d82df4e7-e480-4c9f-99f2-9bf7eacf83df\") "
Jul 30 21:40:17.419342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.419105    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache" (OuterVolumeSpecName: "model-cache") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:17.419342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.419116    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home" (OuterVolumeSpecName: "home") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:17.419342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.419309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:17.419342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.419321    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.419505 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.419366    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.421379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.421347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm" (OuterVolumeSpecName: "dshm") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:40:17.421379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.421348    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:40:17.421522 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.421350    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29" (OuterVolumeSpecName: "kube-api-access-sjm29") pod "d82df4e7-e480-4c9f-99f2-9bf7eacf83df" (UID: "d82df4e7-e480-4c9f-99f2-9bf7eacf83df"). InnerVolumeSpecName "kube-api-access-sjm29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:17.520552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.520512    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.520552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.520546    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjm29\" (UniqueName: \"kubernetes.io/projected/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-kube-api-access-sjm29\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.520552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.520558    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.520805 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.520589    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d82df4e7-e480-4c9f-99f2-9bf7eacf83df-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:17.790879 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.790848    2583 generic.go:356] "Generic (PLEG): container finished" podID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" containerID="d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25" exitCode=0
Jul 30 21:40:17.791086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.790888    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"
Jul 30 21:40:17.791086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.790915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" event={"ID":"d82df4e7-e480-4c9f-99f2-9bf7eacf83df","Type":"ContainerDied","Data":"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"}
Jul 30 21:40:17.791086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.790944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2" event={"ID":"d82df4e7-e480-4c9f-99f2-9bf7eacf83df","Type":"ContainerDied","Data":"540167c987fc1686cd2f48c840dcc83d4d0492d3e04954f698fcfa1eef0453cf"}
Jul 30 21:40:17.791086 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.790960    2583 scope.go:117] "RemoveContainer" containerID="d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"
Jul 30 21:40:17.800577 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.800544    2583 scope.go:117] "RemoveContainer" containerID="d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"
Jul 30 21:40:17.800864 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:17.800844    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25\": container with ID starting with d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25 not found: ID does not exist" containerID="d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"
Jul 30 21:40:17.800923 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.800872    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25"} err="failed to get container status \"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25\": rpc error: code = NotFound desc = could not find container \"d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25\": container with ID starting with d4d0f69f38826e0573fded61dc93becff62cc6ff147d127faaff0f191e9d5c25 not found: ID does not exist"
Jul 30 21:40:17.811326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.811304    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:40:17.816360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:17.816340    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e79f82/test-model-e79f82-kserve-7db78667d7-tc6z2"]
Jul 30 21:40:18.777409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:18.777376    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063a0f04-7470-4c0f-9d27-b51642a7714e" path="/var/lib/kubelet/pods/063a0f04-7470-4c0f-9d27-b51642a7714e/volumes"
Jul 30 21:40:18.777822 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:18.777806    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" path="/var/lib/kubelet/pods/d82df4e7-e480-4c9f-99f2-9bf7eacf83df/volumes"
Jul 30 21:40:19.188206 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.188110    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:19.248808 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.248769    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:19.451598 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.451493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:19.706191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.706098    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:19.808856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.808826    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"]
Jul 30 21:40:19.809253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809221    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2724666b-f878-4711-9e49-1cae6f99093e" containerName="maas-api"
Jul 30 21:40:19.809253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809236    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2724666b-f878-4711-9e49-1cae6f99093e" containerName="maas-api"
Jul 30 21:40:19.809253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809247    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" containerName="main"
Jul 30 21:40:19.809253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809254    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" containerName="main"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809267    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e655149-7318-4be1-82f1-8ebecb8cc97d" containerName="payload-pre-processing"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809277    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e655149-7318-4be1-82f1-8ebecb8cc97d" containerName="payload-pre-processing"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809291    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515a28ad-7804-446f-8c9b-2039982b50d5" containerName="payload-processing"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809297    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="515a28ad-7804-446f-8c9b-2039982b50d5" containerName="payload-processing"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809303    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063a0f04-7470-4c0f-9d27-b51642a7714e" containerName="main"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809308    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="063a0f04-7470-4c0f-9d27-b51642a7714e" containerName="main"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809373    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82df4e7-e480-4c9f-99f2-9bf7eacf83df" containerName="main"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809384    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="515a28ad-7804-446f-8c9b-2039982b50d5" containerName="payload-processing"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809393    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2724666b-f878-4711-9e49-1cae6f99093e" containerName="maas-api"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809401    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="063a0f04-7470-4c0f-9d27-b51642a7714e" containerName="main"
Jul 30 21:40:19.809413 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.809408    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e655149-7318-4be1-82f1-8ebecb8cc97d" containerName="payload-pre-processing"
Jul 30 21:40:19.813659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.813635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:19.815754 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.815734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:40:19.819960 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.819940    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"]
Jul 30 21:40:19.943495 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:19.943458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt\" (UID: \"64c265cd-69f6-4187-b126-84e14d0df8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:20.044818 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.044787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt\" (UID: \"64c265cd-69f6-4187-b126-84e14d0df8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:20.045438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.045417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt\" (UID: \"64c265cd-69f6-4187-b126-84e14d0df8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:20.126654 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.126621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:20.262004 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.261978    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"]
Jul 30 21:40:20.264724 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:40:20.264697    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c265cd_69f6_4187_b126_84e14d0df8e6.slice/crio-5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a WatchSource:0}: Error finding container 5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a: Status 404 returned error can't find the container with id 5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a
Jul 30 21:40:20.808687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.808648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt" event={"ID":"64c265cd-69f6-4187-b126-84e14d0df8e6","Type":"ContainerStarted","Data":"3cae86b82b236bad34bb95bcb90e9513966c5c0d66db5c2d7825d5d883aacf28"}
Jul 30 21:40:20.808872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.808700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt" event={"ID":"64c265cd-69f6-4187-b126-84e14d0df8e6","Type":"ContainerStarted","Data":"5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a"}
Jul 30 21:40:20.822261 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:20.822000    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt" podStartSLOduration=1.8219834320000001 podStartE2EDuration="1.821983432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:40:20.821658747 +0000 UTC m=+2592.587620626" watchObservedRunningTime="2026-07-30 21:40:20.821983432 +0000 UTC m=+2592.587945310"
Jul 30 21:40:21.815007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:21.814968    2583 generic.go:356] "Generic (PLEG): container finished" podID="64c265cd-69f6-4187-b126-84e14d0df8e6" containerID="3cae86b82b236bad34bb95bcb90e9513966c5c0d66db5c2d7825d5d883aacf28" exitCode=0
Jul 30 21:40:21.815007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:21.815009    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt" event={"ID":"64c265cd-69f6-4187-b126-84e14d0df8e6","Type":"ContainerDied","Data":"3cae86b82b236bad34bb95bcb90e9513966c5c0d66db5c2d7825d5d883aacf28"}
Jul 30 21:40:22.958236 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:22.958209    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:22.964995 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:22.964971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca\") pod \"64c265cd-69f6-4187-b126-84e14d0df8e6\" (UID: \"64c265cd-69f6-4187-b126-84e14d0df8e6\") "
Jul 30 21:40:22.965317 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:22.965298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64c265cd-69f6-4187-b126-84e14d0df8e6" (UID: "64c265cd-69f6-4187-b126-84e14d0df8e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:23.066288 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:23.066248    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64c265cd-69f6-4187-b126-84e14d0df8e6-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:23.825449 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:23.825415    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"
Jul 30 21:40:23.825649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:23.825463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt" event={"ID":"64c265cd-69f6-4187-b126-84e14d0df8e6","Type":"ContainerDied","Data":"5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a"}
Jul 30 21:40:23.825649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:23.825510    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a3f0e1de6fb07fc2502f126ca19f817e934b0d7e588f1eba25ead16328d5a7a"
Jul 30 21:40:30.221200 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.221117    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:40:30.221645 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.221304    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" podUID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" containerName="maas-api" containerID="cri-o://4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073" gracePeriod=30
Jul 30 21:40:30.249550 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.249514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:30.276473 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.276442    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:40:30.276700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.276657    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" podUID="3531cb9d-f507-406e-9fb6-9129731bba7d" containerName="payload-processing" containerID="cri-o://1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95" gracePeriod=30
Jul 30 21:40:30.286313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.286272    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:40:30.287142 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.287113    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" podUID="b0d14139-584e-42a3-939c-73f2c8949ea6" containerName="payload-pre-processing" containerID="cri-o://e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4" gracePeriod=30
Jul 30 21:40:30.317280 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.317242    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:30.392521 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.392493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:30.425306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.425264    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"]
Jul 30 21:40:30.428697 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.428642    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e79f82-5bvxt"]
Jul 30 21:40:30.454438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.454395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:30.588322 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.588296    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:40:30.592008 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.591942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:40:30.627532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.627499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume\") pod \"3531cb9d-f507-406e-9fb6-9129731bba7d\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") "
Jul 30 21:40:30.627714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.627551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume\") pod \"b0d14139-584e-42a3-939c-73f2c8949ea6\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") "
Jul 30 21:40:30.627714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.627610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcdwg\" (UniqueName: \"kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg\") pod \"b0d14139-584e-42a3-939c-73f2c8949ea6\" (UID: \"b0d14139-584e-42a3-939c-73f2c8949ea6\") "
Jul 30 21:40:30.628068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.628037    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3531cb9d-f507-406e-9fb6-9129731bba7d" (UID: "3531cb9d-f507-406e-9fb6-9129731bba7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:30.628068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.628061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b0d14139-584e-42a3-939c-73f2c8949ea6" (UID: "b0d14139-584e-42a3-939c-73f2c8949ea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:30.628424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.628110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf\") pod \"3531cb9d-f507-406e-9fb6-9129731bba7d\" (UID: \"3531cb9d-f507-406e-9fb6-9129731bba7d\") "
Jul 30 21:40:30.628424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.628400    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3531cb9d-f507-406e-9fb6-9129731bba7d-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.628541 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.628424    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0d14139-584e-42a3-939c-73f2c8949ea6-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.630438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.630414    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf" (OuterVolumeSpecName: "kube-api-access-2gjsf") pod "3531cb9d-f507-406e-9fb6-9129731bba7d" (UID: "3531cb9d-f507-406e-9fb6-9129731bba7d"). InnerVolumeSpecName "kube-api-access-2gjsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:30.630583 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.630435    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg" (OuterVolumeSpecName: "kube-api-access-wcdwg") pod "b0d14139-584e-42a3-939c-73f2c8949ea6" (UID: "b0d14139-584e-42a3-939c-73f2c8949ea6"). InnerVolumeSpecName "kube-api-access-wcdwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:30.675228 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.675204    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:40:30.728970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.728943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca\") pod \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") "
Jul 30 21:40:30.729159 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.728993    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls\") pod \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") "
Jul 30 21:40:30.729159 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.729038    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jqjs\" (UniqueName: \"kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs\") pod \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\" (UID: \"3de8097a-702b-4ff0-8eb7-614d7799bb3a\") "
Jul 30 21:40:30.729253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.729193    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcdwg\" (UniqueName: \"kubernetes.io/projected/b0d14139-584e-42a3-939c-73f2c8949ea6-kube-api-access-wcdwg\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.729253 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.729211    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjsf\" (UniqueName: \"kubernetes.io/projected/3531cb9d-f507-406e-9fb6-9129731bba7d-kube-api-access-2gjsf\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.729346 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.729293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3de8097a-702b-4ff0-8eb7-614d7799bb3a" (UID: "3de8097a-702b-4ff0-8eb7-614d7799bb3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:40:30.731144 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.731121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs" (OuterVolumeSpecName: "kube-api-access-7jqjs") pod "3de8097a-702b-4ff0-8eb7-614d7799bb3a" (UID: "3de8097a-702b-4ff0-8eb7-614d7799bb3a"). InnerVolumeSpecName "kube-api-access-7jqjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:40:30.731227 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.731190    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3de8097a-702b-4ff0-8eb7-614d7799bb3a" (UID: "3de8097a-702b-4ff0-8eb7-614d7799bb3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:40:30.777449 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.777369    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64c265cd-69f6-4187-b126-84e14d0df8e6" path="/var/lib/kubelet/pods/64c265cd-69f6-4187-b126-84e14d0df8e6/volumes"
Jul 30 21:40:30.815172 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.815137    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:30.830240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.830215    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de8097a-702b-4ff0-8eb7-614d7799bb3a-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.830240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.830239    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de8097a-702b-4ff0-8eb7-614d7799bb3a-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.830378 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.830248    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jqjs\" (UniqueName: \"kubernetes.io/projected/3de8097a-702b-4ff0-8eb7-614d7799bb3a-kube-api-access-7jqjs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:40:30.860542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.860506    2583 generic.go:356] "Generic (PLEG): container finished" podID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" containerID="4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073" exitCode=0
Jul 30 21:40:30.860718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.860553    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"
Jul 30 21:40:30.860718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.860590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" event={"ID":"3de8097a-702b-4ff0-8eb7-614d7799bb3a","Type":"ContainerDied","Data":"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"}
Jul 30 21:40:30.860718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.860643    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9" event={"ID":"3de8097a-702b-4ff0-8eb7-614d7799bb3a","Type":"ContainerDied","Data":"74653f0ed7a031554d847a7a71cb3f8e825f8f39b049ba07045e66c076ac867c"}
Jul 30 21:40:30.860718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.860672    2583 scope.go:117] "RemoveContainer" containerID="4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"
Jul 30 21:40:30.861949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.861919    2583 generic.go:356] "Generic (PLEG): container finished" podID="3531cb9d-f507-406e-9fb6-9129731bba7d" containerID="1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95" exitCode=0
Jul 30 21:40:30.862035 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.861949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" event={"ID":"3531cb9d-f507-406e-9fb6-9129731bba7d","Type":"ContainerDied","Data":"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"}
Jul 30 21:40:30.862035 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.862005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76" event={"ID":"3531cb9d-f507-406e-9fb6-9129731bba7d","Type":"ContainerDied","Data":"8b4a615694caaf25ca0a3b9ce3ec90ed38a4718347040d1f6f916986d03b4b2c"}
Jul 30 21:40:30.862035 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.861958    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"
Jul 30 21:40:30.863587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.863536    2583 generic.go:356] "Generic (PLEG): container finished" podID="b0d14139-584e-42a3-939c-73f2c8949ea6" containerID="e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4" exitCode=0
Jul 30 21:40:30.863708 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.863582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" event={"ID":"b0d14139-584e-42a3-939c-73f2c8949ea6","Type":"ContainerDied","Data":"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"}
Jul 30 21:40:30.863708 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.863604    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"
Jul 30 21:40:30.863708 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.863647    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl" event={"ID":"b0d14139-584e-42a3-939c-73f2c8949ea6","Type":"ContainerDied","Data":"46d68b921477ff64ae2b348850f30f1d4c15c76bdec0585ddea34315d50a5457"}
Jul 30 21:40:30.872581 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.872546    2583 scope.go:117] "RemoveContainer" containerID="4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"
Jul 30 21:40:30.872878 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:30.872859    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073\": container with ID starting with 4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073 not found: ID does not exist" containerID="4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"
Jul 30 21:40:30.872931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.872887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073"} err="failed to get container status \"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073\": rpc error: code = NotFound desc = could not find container \"4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073\": container with ID starting with 4e0b94a431579a1c509d14e7c6ccd9029983098f5a9082605154a21234cb8073 not found: ID does not exist"
Jul 30 21:40:30.872931 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.872904    2583 scope.go:117] "RemoveContainer" containerID="1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"
Jul 30 21:40:30.880023 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.879999    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:40:30.880886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.880867    2583 scope.go:117] "RemoveContainer" containerID="1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"
Jul 30 21:40:30.881162 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:30.881145    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95\": container with ID starting with 1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95 not found: ID does not exist" containerID="1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"
Jul 30 21:40:30.881218 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.881169    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95"} err="failed to get container status \"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95\": rpc error: code = NotFound desc = could not find container \"1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95\": container with ID starting with 1495e610dbce14a5cee0b8e9522a916b49ba55eb19147e52522505af8e174d95 not found: ID does not exist"
Jul 30 21:40:30.881218 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.881184    2583 scope.go:117] "RemoveContainer" containerID="e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"
Jul 30 21:40:30.883521 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.883503    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e79f82-688795bb9d-7ql76"]
Jul 30 21:40:30.891096 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.891053    2583 scope.go:117] "RemoveContainer" containerID="e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"
Jul 30 21:40:30.891417 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:30.891358    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4\": container with ID starting with e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4 not found: ID does not exist" containerID="e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"
Jul 30 21:40:30.891417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.891391    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4"} err="failed to get container status \"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4\": rpc error: code = NotFound desc = could not find container \"e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4\": container with ID starting with e5023527ff1532f5704371900d80b673972abd32847c1f426be2b9cad8e32db4 not found: ID does not exist"
Jul 30 21:40:30.892956 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.892935    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:40:30.898011 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.897991    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e79f82-74dc8796cf-sk2n9"]
Jul 30 21:40:30.906025 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.906001    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:40:30.911127 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:30.911108    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e79f82-75dccdbf56-ktntl"]
Jul 30 21:40:32.778442 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:32.778409    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3531cb9d-f507-406e-9fb6-9129731bba7d" path="/var/lib/kubelet/pods/3531cb9d-f507-406e-9fb6-9129731bba7d/volumes"
Jul 30 21:40:32.778919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:32.778824    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" path="/var/lib/kubelet/pods/3de8097a-702b-4ff0-8eb7-614d7799bb3a/volumes"
Jul 30 21:40:32.779118 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:32.779105    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0d14139-584e-42a3-939c-73f2c8949ea6" path="/var/lib/kubelet/pods/b0d14139-584e-42a3-939c-73f2c8949ea6/volumes"
Jul 30 21:40:36.623692 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:36.623653    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:36.684503 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:36.684459    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:36.898444 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:36.898362    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:37.118936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:37.118906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:37.459898 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:37.459862    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:37.762561 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:37.762525    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:38.100610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:38.100502    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:40.611327 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:40.611281    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:40.710122 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:40.710074    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:41.697009 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.696972    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:40:41.697447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697430    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" containerName="maas-api"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697449    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" containerName="maas-api"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697476    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64c265cd-69f6-4187-b126-84e14d0df8e6" containerName="revoke-keys"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697484    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="64c265cd-69f6-4187-b126-84e14d0df8e6" containerName="revoke-keys"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697497    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3531cb9d-f507-406e-9fb6-9129731bba7d" containerName="payload-processing"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697505    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3531cb9d-f507-406e-9fb6-9129731bba7d" containerName="payload-processing"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697518    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0d14139-584e-42a3-939c-73f2c8949ea6" containerName="payload-pre-processing"
Jul 30 21:40:41.697535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697526    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0d14139-584e-42a3-939c-73f2c8949ea6" containerName="payload-pre-processing"
Jul 30 21:40:41.697908 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697646    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="64c265cd-69f6-4187-b126-84e14d0df8e6" containerName="revoke-keys"
Jul 30 21:40:41.697908 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697658    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0d14139-584e-42a3-939c-73f2c8949ea6" containerName="payload-pre-processing"
Jul 30 21:40:41.697908 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697666    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3531cb9d-f507-406e-9fb6-9129731bba7d" containerName="payload-processing"
Jul 30 21:40:41.697908 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.697673    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de8097a-702b-4ff0-8eb7-614d7799bb3a" containerName="maas-api"
Jul 30 21:40:41.702611 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.702591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.704301 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.704281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-db28eb-serving-cert\""
Jul 30 21:40:41.704416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.704342    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:40:41.711091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.711065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:40:41.733286 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.733252    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:41.819185 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.819155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.819360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.819210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwqzp\" (UniqueName: \"kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.819360 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.819275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.920255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.920216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.920463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.920276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwqzp\" (UniqueName: \"kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.920463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.920337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.920463 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:41.920379    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-serving-cert: secret "maas-api-e2e-api-a-db28eb-serving-cert" not found
Jul 30 21:40:41.920463 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:40:41.920459    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls podName:d84bf3d3-3266-461f-b0a2-aba175727a02 nodeName:}" failed. No retries permitted until 2026-07-30 21:40:42.4204415 +0000 UTC m=+2614.186403357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls") pod "maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" (UID: "d84bf3d3-3266-461f-b0a2-aba175727a02") : secret "maas-api-e2e-api-a-db28eb-serving-cert" not found
Jul 30 21:40:41.920953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.920934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:41.928210 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:41.928188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwqzp\" (UniqueName: \"kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:42.196500 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.196440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:42.319662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.319627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:42.424307 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.424271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:42.425271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.425233    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:40:42.427332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.427307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") pod \"maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:42.428038 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.428015    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:40:42.431454 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.431435    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.433243 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.433224    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-db28eb-dockercfg-95l5z\""
Jul 30 21:40:42.433365 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.433225    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-db28eb\""
Jul 30 21:40:42.440649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.440623    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:40:42.463895 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.463825    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:40:42.467025 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.467008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.475005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.474489    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:40:42.525472 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.525436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.525685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.525487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srdq\" (UniqueName: \"kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.525685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.525614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh4jx\" (UniqueName: \"kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.525789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.525681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.614934 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.614900    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:42.626539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.626502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh4jx\" (UniqueName: \"kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.626703 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.626623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.626703 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.626670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.626855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.626727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srdq\" (UniqueName: \"kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.627871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.627843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.628470 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.628449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.634948 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.634922    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh4jx\" (UniqueName: \"kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx\") pod \"payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") " pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.635103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.635078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srdq\" (UniqueName: \"kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq\") pod \"payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.745264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.745222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:42.757984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.757959    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:40:42.760870 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:40:42.760835    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84bf3d3_3266_461f_b0a2_aba175727a02.slice/crio-dee40eb2e3850d018d9a9ee903ff887ce4895481468322bc409a24521f1fe8e5 WatchSource:0}: Error finding container dee40eb2e3850d018d9a9ee903ff887ce4895481468322bc409a24521f1fe8e5: Status 404 returned error can't find the container with id dee40eb2e3850d018d9a9ee903ff887ce4895481468322bc409a24521f1fe8e5
Jul 30 21:40:42.762487 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.762463    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:40:42.778251 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.778220    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:42.905136 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.905091    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:40:42.910039 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:40:42.910009    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d352f2d_a021_409e_88e9_975fd25a6af9.slice/crio-5c2003f01432ad679c0033ab382acd6d0c9571b242ecc9469ab2fe45c9a66e2d WatchSource:0}: Error finding container 5c2003f01432ad679c0033ab382acd6d0c9571b242ecc9469ab2fe45c9a66e2d: Status 404 returned error can't find the container with id 5c2003f01432ad679c0033ab382acd6d0c9571b242ecc9469ab2fe45c9a66e2d
Jul 30 21:40:42.931809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.931780    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" event={"ID":"0d352f2d-a021-409e-88e9-975fd25a6af9","Type":"ContainerStarted","Data":"5c2003f01432ad679c0033ab382acd6d0c9571b242ecc9469ab2fe45c9a66e2d"}
Jul 30 21:40:42.933482 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.933449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" event={"ID":"d84bf3d3-3266-461f-b0a2-aba175727a02","Type":"ContainerStarted","Data":"dee40eb2e3850d018d9a9ee903ff887ce4895481468322bc409a24521f1fe8e5"}
Jul 30 21:40:42.935103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:42.935077    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:40:42.938765 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:40:42.938733    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed4c5fb_fa5f_4f64_be42_519738065d1b.slice/crio-e2bc38d5429f86e54f64e74025291ee6a77d922be45db53f04be02ce13040c22 WatchSource:0}: Error finding container e2bc38d5429f86e54f64e74025291ee6a77d922be45db53f04be02ce13040c22: Status 404 returned error can't find the container with id e2bc38d5429f86e54f64e74025291ee6a77d922be45db53f04be02ce13040c22
Jul 30 21:40:43.940623 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.940559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" event={"ID":"0d352f2d-a021-409e-88e9-975fd25a6af9","Type":"ContainerStarted","Data":"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"}
Jul 30 21:40:43.941097 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.940709    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:40:43.942091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.942062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" event={"ID":"d84bf3d3-3266-461f-b0a2-aba175727a02","Type":"ContainerStarted","Data":"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"}
Jul 30 21:40:43.942213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.942125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:43.943581 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.943534    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" event={"ID":"aed4c5fb-fa5f-4f64-be42-519738065d1b","Type":"ContainerStarted","Data":"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"}
Jul 30 21:40:43.943687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.943612    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:40:43.943687 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.943632    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" event={"ID":"aed4c5fb-fa5f-4f64-be42-519738065d1b","Type":"ContainerStarted","Data":"e2bc38d5429f86e54f64e74025291ee6a77d922be45db53f04be02ce13040c22"}
Jul 30 21:40:43.954503 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.954462    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" podStartSLOduration=1.9544499850000001 podStartE2EDuration="1.954449985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:40:43.953531039 +0000 UTC m=+2615.719492921" watchObservedRunningTime="2026-07-30 21:40:43.954449985 +0000 UTC m=+2615.720411924"
Jul 30 21:40:43.968003 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.967957    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" podStartSLOduration=2.450430315 podStartE2EDuration="2.967943432s" totalImagesPullingTime="517.513117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:40:43.967009064 +0000 UTC m=+2615.732970942" watchObservedRunningTime="2026-07-30 21:40:43.967943432 +0000 UTC m=+2615.733905362"
Jul 30 21:40:43.982286 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:43.982236    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" podStartSLOduration=1.98222174 podStartE2EDuration="1.98222174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:40:43.980853949 +0000 UTC m=+2615.746815827" watchObservedRunningTime="2026-07-30 21:40:43.98222174 +0000 UTC m=+2615.748183618"
Jul 30 21:40:49.955543 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:49.955515    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:40:54.951160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:54.951112    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.82:9004: connect: connection refused"
Jul 30 21:40:54.951657 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:40:54.951529    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:41:04.950769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:04.950735    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:41:05.592408 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:05.592373    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:06.056305 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:06.056264    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:06.362626 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:06.362500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:06.462557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:06.462523    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:09.880006 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:09.879965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:09.978275 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:09.978234    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:11.151940 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.151901    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:41:11.157499 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.157476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.159310 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.159293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d8e5fa-serving-cert\""
Jul 30 21:41:11.165026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.164996    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:41:11.183266 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.183235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:11.279985 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.279949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.280168 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.279995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.280168 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.280062    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dbx\" (UniqueName: \"kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.380938 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.380896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.381149 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.380954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.381149 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.380997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dbx\" (UniqueName: \"kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.381149 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:41:11.381061    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-serving-cert: secret "maas-api-e2e-api-b-d8e5fa-serving-cert" not found
Jul 30 21:41:11.381149 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:41:11.381130    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls podName:c14c00b8-776f-4c5d-908e-a87116339ab4 nodeName:}" failed. No retries permitted until 2026-07-30 21:41:11.881111752 +0000 UTC m=+2643.647073610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls") pod "maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" (UID: "c14c00b8-776f-4c5d-908e-a87116339ab4") : secret "maas-api-e2e-api-b-d8e5fa-serving-cert" not found
Jul 30 21:41:11.381629 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.381611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.389369 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.389341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dbx\" (UniqueName: \"kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.738628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.738554    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:11.838703 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.838666    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:11.880524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.880489    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:41:11.884005 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.883981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:11.884759 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.884740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.885997 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.885977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d8e5fa-dockercfg-nm6zw\""
Jul 30 21:41:11.886160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.885978    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d8e5fa\""
Jul 30 21:41:11.887885 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.887863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") pod \"maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:11.894531 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.894510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:41:11.917223 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.917195    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:41:11.926232 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.926206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:11.928649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.928610    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:41:11.985591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.985522    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:11.985818 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:11.985641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghjg\" (UniqueName: \"kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.070928 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.070832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:12.087119 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.087079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.087264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.087164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghjg\" (UniqueName: \"kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.087264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.087196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.087264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.087259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.088194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.088150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.096096 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.096073    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghjg\" (UniqueName: \"kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg\") pod \"payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") " pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.188589 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.188349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.188589 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.188438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.189391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.189361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.196501 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.196474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4\") pod \"payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.207938 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.207909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:12.237357 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.237326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:12.365588 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.365538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:41:12.369025 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:41:12.368992    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f55655_5328_40b7_aaa9_15df2acba799.slice/crio-ecdffb0ddc853b616962aba13528f1c976f298440196e7126f7f5067eb417a68 WatchSource:0}: Error finding container ecdffb0ddc853b616962aba13528f1c976f298440196e7126f7f5067eb417a68: Status 404 returned error can't find the container with id ecdffb0ddc853b616962aba13528f1c976f298440196e7126f7f5067eb417a68
Jul 30 21:41:12.395340 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.395309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:41:12.397701 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:41:12.397668    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3622d79f_43ba_484c_9d9b_ed75c2adb449.slice/crio-bece0d72c4b6ba03c10b394d567b79c5ebf141e81ae088572134b2d268c3577d WatchSource:0}: Error finding container bece0d72c4b6ba03c10b394d567b79c5ebf141e81ae088572134b2d268c3577d: Status 404 returned error can't find the container with id bece0d72c4b6ba03c10b394d567b79c5ebf141e81ae088572134b2d268c3577d
Jul 30 21:41:12.423893 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:12.423866    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:41:12.426682 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:41:12.426656    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14c00b8_776f_4c5d_908e_a87116339ab4.slice/crio-ceb43eae07f45fb680374b90b819d955417bc106f560168bd995fdd07041c221 WatchSource:0}: Error finding container ceb43eae07f45fb680374b90b819d955417bc106f560168bd995fdd07041c221: Status 404 returned error can't find the container with id ceb43eae07f45fb680374b90b819d955417bc106f560168bd995fdd07041c221
Jul 30 21:41:13.110749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.110704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" event={"ID":"e9f55655-5328-40b7-aaa9-15df2acba799","Type":"ContainerStarted","Data":"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"}
Jul 30 21:41:13.110936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.110783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" event={"ID":"e9f55655-5328-40b7-aaa9-15df2acba799","Type":"ContainerStarted","Data":"ecdffb0ddc853b616962aba13528f1c976f298440196e7126f7f5067eb417a68"}
Jul 30 21:41:13.110936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.110819    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:13.112504 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.112475    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" event={"ID":"c14c00b8-776f-4c5d-908e-a87116339ab4","Type":"ContainerStarted","Data":"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"}
Jul 30 21:41:13.112666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.112526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" event={"ID":"c14c00b8-776f-4c5d-908e-a87116339ab4","Type":"ContainerStarted","Data":"ceb43eae07f45fb680374b90b819d955417bc106f560168bd995fdd07041c221"}
Jul 30 21:41:13.112666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.112556    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:13.114238 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.114209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" event={"ID":"3622d79f-43ba-484c-9d9b-ed75c2adb449","Type":"ContainerStarted","Data":"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"}
Jul 30 21:41:13.114384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.114263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:13.114384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.114288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" event={"ID":"3622d79f-43ba-484c-9d9b-ed75c2adb449","Type":"ContainerStarted","Data":"bece0d72c4b6ba03c10b394d567b79c5ebf141e81ae088572134b2d268c3577d"}
Jul 30 21:41:13.127816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.127773    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" podStartSLOduration=2.127758651 podStartE2EDuration="2.127758651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:41:13.12585661 +0000 UTC m=+2644.891818490" watchObservedRunningTime="2026-07-30 21:41:13.127758651 +0000 UTC m=+2644.893720529"
Jul 30 21:41:13.141423 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.141366    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" podStartSLOduration=2.141351638 podStartE2EDuration="2.141351638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:41:13.139096369 +0000 UTC m=+2644.905058247" watchObservedRunningTime="2026-07-30 21:41:13.141351638 +0000 UTC m=+2644.907313495"
Jul 30 21:41:13.154080 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:13.154031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" podStartSLOduration=1.634866138 podStartE2EDuration="2.154015284s" totalImagesPullingTime="519.149146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:41:13.152012547 +0000 UTC m=+2644.917974449" watchObservedRunningTime="2026-07-30 21:41:13.154015284 +0000 UTC m=+2644.919977162"
Jul 30 21:41:19.125549 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:19.125519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:41:24.122535 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:24.122504    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:41:24.123012 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:24.122612    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:41:34.797585 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:34.797528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:37.756468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.756431    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"]
Jul 30 21:41:37.760142 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.760120    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:37.762028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.761995    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:41:37.766811 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.766779    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"]
Jul 30 21:41:37.803103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.803068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db28eb-sctss\" (UID: \"fd72b64e-6fed-4853-ac82-a9e6e210a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:37.904443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.904402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db28eb-sctss\" (UID: \"fd72b64e-6fed-4853-ac82-a9e6e210a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:37.905077 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:37.905049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db28eb-sctss\" (UID: \"fd72b64e-6fed-4853-ac82-a9e6e210a8c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:38.073069 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:38.072982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:38.208441 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:38.208283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"]
Jul 30 21:41:38.211288 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:41:38.211262    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd72b64e_6fed_4853_ac82_a9e6e210a8c1.slice/crio-92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b WatchSource:0}: Error finding container 92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b: Status 404 returned error can't find the container with id 92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b
Jul 30 21:41:38.259900 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:38.259871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss" event={"ID":"fd72b64e-6fed-4853-ac82-a9e6e210a8c1","Type":"ContainerStarted","Data":"92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b"}
Jul 30 21:41:39.267268 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:39.267226    2583 generic.go:356] "Generic (PLEG): container finished" podID="fd72b64e-6fed-4853-ac82-a9e6e210a8c1" containerID="1ba150a21cbb6f3aacec2325a45fbb0597d1d415a70ee868a9a02cf53bf23b1a" exitCode=0
Jul 30 21:41:39.267733 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:39.267292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss" event={"ID":"fd72b64e-6fed-4853-ac82-a9e6e210a8c1","Type":"ContainerDied","Data":"1ba150a21cbb6f3aacec2325a45fbb0597d1d415a70ee868a9a02cf53bf23b1a"}
Jul 30 21:41:40.415823 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:40.415797    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:40.530412 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:40.530324    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca\") pod \"fd72b64e-6fed-4853-ac82-a9e6e210a8c1\" (UID: \"fd72b64e-6fed-4853-ac82-a9e6e210a8c1\") "
Jul 30 21:41:40.530704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:40.530674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd72b64e-6fed-4853-ac82-a9e6e210a8c1" (UID: "fd72b64e-6fed-4853-ac82-a9e6e210a8c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:41:40.631066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:40.631033    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd72b64e-6fed-4853-ac82-a9e6e210a8c1-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:41.279263 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:41.279233    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"
Jul 30 21:41:41.279444 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:41.279273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss" event={"ID":"fd72b64e-6fed-4853-ac82-a9e6e210a8c1","Type":"ContainerDied","Data":"92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b"}
Jul 30 21:41:41.279444 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:41.279322    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92de090816d87c6ab61c3e4fab7ea6195b566c4ea24c3999aa9b848cb1cb926b"
Jul 30 21:41:53.267858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.267803    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:41:53.268397 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.268039    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" podUID="d84bf3d3-3266-461f-b0a2-aba175727a02" containerName="maas-api" containerID="cri-o://dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0" gracePeriod=30
Jul 30 21:41:53.295923 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.295895    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:53.321230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.321193    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:41:53.321447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.321403    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" podUID="aed4c5fb-fa5f-4f64-be42-519738065d1b" containerName="payload-processing" containerID="cri-o://d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e" gracePeriod=30
Jul 30 21:41:53.332531 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.332503    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:41:53.332802 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.332756    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerName="payload-pre-processing" containerID="cri-o://bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef" gracePeriod=30
Jul 30 21:41:53.395140 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.395064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:53.464440 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.464399    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"]
Jul 30 21:41:53.469387 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.469348    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:53.469582 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.469405    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db28eb-sctss"]
Jul 30 21:41:53.528341 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.528235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:53.619364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.619337    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:41:53.622054 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.622030    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:41:53.637179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.637156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:41:53.741124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") pod \"d84bf3d3-3266-461f-b0a2-aba175727a02\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") "
Jul 30 21:41:53.741124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5srdq\" (UniqueName: \"kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq\") pod \"0d352f2d-a021-409e-88e9-975fd25a6af9\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") "
Jul 30 21:41:53.741380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwqzp\" (UniqueName: \"kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp\") pod \"d84bf3d3-3266-461f-b0a2-aba175727a02\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") "
Jul 30 21:41:53.741380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741195    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume\") pod \"0d352f2d-a021-409e-88e9-975fd25a6af9\" (UID: \"0d352f2d-a021-409e-88e9-975fd25a6af9\") "
Jul 30 21:41:53.741380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca\") pod \"d84bf3d3-3266-461f-b0a2-aba175727a02\" (UID: \"d84bf3d3-3266-461f-b0a2-aba175727a02\") "
Jul 30 21:41:53.741380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741246    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh4jx\" (UniqueName: \"kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx\") pod \"aed4c5fb-fa5f-4f64-be42-519738065d1b\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") "
Jul 30 21:41:53.741380 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume\") pod \"aed4c5fb-fa5f-4f64-be42-519738065d1b\" (UID: \"aed4c5fb-fa5f-4f64-be42-519738065d1b\") "
Jul 30 21:41:53.741705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741638    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d352f2d-a021-409e-88e9-975fd25a6af9" (UID: "0d352f2d-a021-409e-88e9-975fd25a6af9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:41:53.741761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741720    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d84bf3d3-3266-461f-b0a2-aba175727a02" (UID: "d84bf3d3-3266-461f-b0a2-aba175727a02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:41:53.741843 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.741817    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aed4c5fb-fa5f-4f64-be42-519738065d1b" (UID: "aed4c5fb-fa5f-4f64-be42-519738065d1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:41:53.743611 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.743558    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp" (OuterVolumeSpecName: "kube-api-access-fwqzp") pod "d84bf3d3-3266-461f-b0a2-aba175727a02" (UID: "d84bf3d3-3266-461f-b0a2-aba175727a02"). InnerVolumeSpecName "kube-api-access-fwqzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:41:53.743736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.743628    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx" (OuterVolumeSpecName: "kube-api-access-gh4jx") pod "aed4c5fb-fa5f-4f64-be42-519738065d1b" (UID: "aed4c5fb-fa5f-4f64-be42-519738065d1b"). InnerVolumeSpecName "kube-api-access-gh4jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:41:53.743736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.743647    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d84bf3d3-3266-461f-b0a2-aba175727a02" (UID: "d84bf3d3-3266-461f-b0a2-aba175727a02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:41:53.743736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.743671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq" (OuterVolumeSpecName: "kube-api-access-5srdq") pod "0d352f2d-a021-409e-88e9-975fd25a6af9" (UID: "0d352f2d-a021-409e-88e9-975fd25a6af9"). InnerVolumeSpecName "kube-api-access-5srdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:41:53.813364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.813266    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:53.842848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842817    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84bf3d3-3266-461f-b0a2-aba175727a02-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.842848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842844    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh4jx\" (UniqueName: \"kubernetes.io/projected/aed4c5fb-fa5f-4f64-be42-519738065d1b-kube-api-access-gh4jx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.842848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842854    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed4c5fb-fa5f-4f64-be42-519738065d1b-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.843053 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842863    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84bf3d3-3266-461f-b0a2-aba175727a02-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.843053 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5srdq\" (UniqueName: \"kubernetes.io/projected/0d352f2d-a021-409e-88e9-975fd25a6af9-kube-api-access-5srdq\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.843053 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842880    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwqzp\" (UniqueName: \"kubernetes.io/projected/d84bf3d3-3266-461f-b0a2-aba175727a02-kube-api-access-fwqzp\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:53.843053 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:53.842888    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d352f2d-a021-409e-88e9-975fd25a6af9-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:41:54.350425 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.350390    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerID="bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef" exitCode=0
Jul 30 21:41:54.350937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.350435    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"
Jul 30 21:41:54.350937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.350484    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" event={"ID":"0d352f2d-a021-409e-88e9-975fd25a6af9","Type":"ContainerDied","Data":"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"}
Jul 30 21:41:54.350937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.350548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5" event={"ID":"0d352f2d-a021-409e-88e9-975fd25a6af9","Type":"ContainerDied","Data":"5c2003f01432ad679c0033ab382acd6d0c9571b242ecc9469ab2fe45c9a66e2d"}
Jul 30 21:41:54.350937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.350617    2583 scope.go:117] "RemoveContainer" containerID="bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"
Jul 30 21:41:54.352120 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.352097    2583 generic.go:356] "Generic (PLEG): container finished" podID="d84bf3d3-3266-461f-b0a2-aba175727a02" containerID="dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0" exitCode=0
Jul 30 21:41:54.352188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.352140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" event={"ID":"d84bf3d3-3266-461f-b0a2-aba175727a02","Type":"ContainerDied","Data":"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"}
Jul 30 21:41:54.352188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.352156    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"
Jul 30 21:41:54.352300 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.352194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb" event={"ID":"d84bf3d3-3266-461f-b0a2-aba175727a02","Type":"ContainerDied","Data":"dee40eb2e3850d018d9a9ee903ff887ce4895481468322bc409a24521f1fe8e5"}
Jul 30 21:41:54.353838 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.353812    2583 generic.go:356] "Generic (PLEG): container finished" podID="aed4c5fb-fa5f-4f64-be42-519738065d1b" containerID="d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e" exitCode=0
Jul 30 21:41:54.353942 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.353837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" event={"ID":"aed4c5fb-fa5f-4f64-be42-519738065d1b","Type":"ContainerDied","Data":"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"}
Jul 30 21:41:54.353942 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.353864    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"
Jul 30 21:41:54.353942 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.353884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd" event={"ID":"aed4c5fb-fa5f-4f64-be42-519738065d1b","Type":"ContainerDied","Data":"e2bc38d5429f86e54f64e74025291ee6a77d922be45db53f04be02ce13040c22"}
Jul 30 21:41:54.363176 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.362408    2583 scope.go:117] "RemoveContainer" containerID="bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"
Jul 30 21:41:54.363306 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:41:54.363256    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef\": container with ID starting with bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef not found: ID does not exist" containerID="bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"
Jul 30 21:41:54.363367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.363297    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef"} err="failed to get container status \"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef\": rpc error: code = NotFound desc = could not find container \"bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef\": container with ID starting with bf1c35f4c4498695e30148daf0c4dbaff93dcf73cb2d20fb39c0bcba47d6b3ef not found: ID does not exist"
Jul 30 21:41:54.363367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.363321    2583 scope.go:117] "RemoveContainer" containerID="dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"
Jul 30 21:41:54.378175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.378149    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:41:54.379172 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.379150    2583 scope.go:117] "RemoveContainer" containerID="dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"
Jul 30 21:41:54.379654 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:41:54.379560    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0\": container with ID starting with dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0 not found: ID does not exist" containerID="dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"
Jul 30 21:41:54.379769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.379666    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0"} err="failed to get container status \"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0\": rpc error: code = NotFound desc = could not find container \"dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0\": container with ID starting with dd6d3479b0e8f97e4c6fbda095dff0a44139907da5f84ed02ad39c665a975ce0 not found: ID does not exist"
Jul 30 21:41:54.379769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.379690    2583 scope.go:117] "RemoveContainer" containerID="d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"
Jul 30 21:41:54.381222 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.381159    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db28eb-6fcfc97d84-6sxk5"]
Jul 30 21:41:54.388284 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.388264    2583 scope.go:117] "RemoveContainer" containerID="d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"
Jul 30 21:41:54.388666 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:41:54.388646    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e\": container with ID starting with d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e not found: ID does not exist" containerID="d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"
Jul 30 21:41:54.388776 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.388673    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e"} err="failed to get container status \"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e\": rpc error: code = NotFound desc = could not find container \"d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e\": container with ID starting with d73ed43992ac652136d269b49f7b4c44bfc12987d679c7fce07315323ade906e not found: ID does not exist"
Jul 30 21:41:54.389271 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.389252    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:41:54.393243 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.393218    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-db28eb-6dd8f78c7f-vdfwb"]
Jul 30 21:41:54.408933 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.408908    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:41:54.414140 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.414120    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db28eb-7d49666f46-rfbsd"]
Jul 30 21:41:54.778919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.778882    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" path="/var/lib/kubelet/pods/0d352f2d-a021-409e-88e9-975fd25a6af9/volumes"
Jul 30 21:41:54.779458 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.779439    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed4c5fb-fa5f-4f64-be42-519738065d1b" path="/var/lib/kubelet/pods/aed4c5fb-fa5f-4f64-be42-519738065d1b/volumes"
Jul 30 21:41:54.779994 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.779976    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d84bf3d3-3266-461f-b0a2-aba175727a02" path="/var/lib/kubelet/pods/d84bf3d3-3266-461f-b0a2-aba175727a02/volumes"
Jul 30 21:41:54.780437 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:54.780418    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd72b64e-6fed-4853-ac82-a9e6e210a8c1" path="/var/lib/kubelet/pods/fd72b64e-6fed-4853-ac82-a9e6e210a8c1/volumes"
Jul 30 21:41:59.532306 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:59.532271    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:59.633103 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:59.633051    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:59.788991 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:59.788870    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:41:59.996280 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:41:59.996241    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:00.112248 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112156    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"]
Jul 30 21:42:00.112554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112538    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd72b64e-6fed-4853-ac82-a9e6e210a8c1" containerName="revoke-keys"
Jul 30 21:42:00.112554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112554    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd72b64e-6fed-4853-ac82-a9e6e210a8c1" containerName="revoke-keys"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112584    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d84bf3d3-3266-461f-b0a2-aba175727a02" containerName="maas-api"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112593    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d84bf3d3-3266-461f-b0a2-aba175727a02" containerName="maas-api"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed4c5fb-fa5f-4f64-be42-519738065d1b" containerName="payload-processing"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112612    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed4c5fb-fa5f-4f64-be42-519738065d1b" containerName="payload-processing"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112626    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerName="payload-pre-processing"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112632    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerName="payload-pre-processing"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112698    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd72b64e-6fed-4853-ac82-a9e6e210a8c1" containerName="revoke-keys"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112706    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed4c5fb-fa5f-4f64-be42-519738065d1b" containerName="payload-processing"
Jul 30 21:42:00.112713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112713    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d352f2d-a021-409e-88e9-975fd25a6af9" containerName="payload-pre-processing"
Jul 30 21:42:00.112973 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.112718    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d84bf3d3-3266-461f-b0a2-aba175727a02" containerName="maas-api"
Jul 30 21:42:00.117113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.117095    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:00.118879 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.118860    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:42:00.121704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.121683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"]
Jul 30 21:42:00.202637 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.202596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv\" (UID: \"135e9b21-cf78-4690-afc7-428eef1e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:00.303323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.303285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv\" (UID: \"135e9b21-cf78-4690-afc7-428eef1e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:00.303980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.303952    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv\" (UID: \"135e9b21-cf78-4690-afc7-428eef1e8c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:00.430055 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.429973    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:00.567693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:00.567664    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"]
Jul 30 21:42:00.570610 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:00.570559    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135e9b21_cf78_4690_afc7_428eef1e8c05.slice/crio-9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c WatchSource:0}: Error finding container 9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c: Status 404 returned error can't find the container with id 9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c
Jul 30 21:42:01.393123 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:01.393083    2583 generic.go:356] "Generic (PLEG): container finished" podID="135e9b21-cf78-4690-afc7-428eef1e8c05" containerID="60a34cd003c1ce624f9b5f1807fc519d11d884b4524ca97b7c74ec84f6ec011f" exitCode=0
Jul 30 21:42:01.393278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:01.393162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv" event={"ID":"135e9b21-cf78-4690-afc7-428eef1e8c05","Type":"ContainerDied","Data":"60a34cd003c1ce624f9b5f1807fc519d11d884b4524ca97b7c74ec84f6ec011f"}
Jul 30 21:42:01.393278 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:01.393215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv" event={"ID":"135e9b21-cf78-4690-afc7-428eef1e8c05","Type":"ContainerStarted","Data":"9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c"}
Jul 30 21:42:02.535217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:02.535192    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:02.624586 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:02.624528    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca\") pod \"135e9b21-cf78-4690-afc7-428eef1e8c05\" (UID: \"135e9b21-cf78-4690-afc7-428eef1e8c05\") "
Jul 30 21:42:02.624899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:02.624869    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "135e9b21-cf78-4690-afc7-428eef1e8c05" (UID: "135e9b21-cf78-4690-afc7-428eef1e8c05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:42:02.726073 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:02.726040    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/135e9b21-cf78-4690-afc7-428eef1e8c05-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:03.404515 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:03.404477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv" event={"ID":"135e9b21-cf78-4690-afc7-428eef1e8c05","Type":"ContainerDied","Data":"9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c"}
Jul 30 21:42:03.404713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:03.404489    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"
Jul 30 21:42:03.404713 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:03.404583    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b271071e273e796030544459da4dce14e3d6d40a533bea7aa92fde240227f7c"
Jul 30 21:42:09.191490 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:09.191465    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:42:09.204962 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:09.204933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:42:09.338816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:09.338778    2583 scope.go:117] "RemoveContainer" containerID="141d34c1d6e5daa48d47c53a77cb1fc4826c7769c73e78a0885812e9dba55b16"
Jul 30 21:42:15.560847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.560809    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:42:15.561681 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.561615    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" podUID="c14c00b8-776f-4c5d-908e-a87116339ab4" containerName="maas-api" containerID="cri-o://54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f" gracePeriod=30
Jul 30 21:42:15.581084 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.581056    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:15.613358 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.613325    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:42:15.613581 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.613539    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" podUID="e9f55655-5328-40b7-aaa9-15df2acba799" containerName="payload-processing" containerID="cri-o://3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950" gracePeriod=30
Jul 30 21:42:15.621992 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.621962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:42:15.622231 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.622205    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" podUID="3622d79f-43ba-484c-9d9b-ed75c2adb449" containerName="payload-pre-processing" containerID="cri-o://c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b" gracePeriod=30
Jul 30 21:42:15.652933 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.652902    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:15.713613 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.713560    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:15.757762 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.757722    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"]
Jul 30 21:42:15.761979 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.761944    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d8e5fa-bxbnv"]
Jul 30 21:42:15.774618 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.774586    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:15.893033 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.893007    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:42:15.923345 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.923316    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:42:15.926937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:15.926909    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:42:16.034256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034224    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") pod \"c14c00b8-776f-4c5d-908e-a87116339ab4\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ghjg\" (UniqueName: \"kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg\") pod \"e9f55655-5328-40b7-aaa9-15df2acba799\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034302    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca\") pod \"c14c00b8-776f-4c5d-908e-a87116339ab4\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume\") pod \"3622d79f-43ba-484c-9d9b-ed75c2adb449\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2dbx\" (UniqueName: \"kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx\") pod \"c14c00b8-776f-4c5d-908e-a87116339ab4\" (UID: \"c14c00b8-776f-4c5d-908e-a87116339ab4\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume\") pod \"e9f55655-5328-40b7-aaa9-15df2acba799\" (UID: \"e9f55655-5328-40b7-aaa9-15df2acba799\") "
Jul 30 21:42:16.034479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034403    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4\") pod \"3622d79f-43ba-484c-9d9b-ed75c2adb449\" (UID: \"3622d79f-43ba-484c-9d9b-ed75c2adb449\") "
Jul 30 21:42:16.035342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034788    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3622d79f-43ba-484c-9d9b-ed75c2adb449" (UID: "3622d79f-43ba-484c-9d9b-ed75c2adb449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:42:16.035342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c14c00b8-776f-4c5d-908e-a87116339ab4" (UID: "c14c00b8-776f-4c5d-908e-a87116339ab4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:42:16.035342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.034847    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9f55655-5328-40b7-aaa9-15df2acba799" (UID: "e9f55655-5328-40b7-aaa9-15df2acba799"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:42:16.036747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.036715    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c14c00b8-776f-4c5d-908e-a87116339ab4" (UID: "c14c00b8-776f-4c5d-908e-a87116339ab4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:42:16.036855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.036800    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx" (OuterVolumeSpecName: "kube-api-access-h2dbx") pod "c14c00b8-776f-4c5d-908e-a87116339ab4" (UID: "c14c00b8-776f-4c5d-908e-a87116339ab4"). InnerVolumeSpecName "kube-api-access-h2dbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:42:16.036855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.036820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4" (OuterVolumeSpecName: "kube-api-access-7ktj4") pod "3622d79f-43ba-484c-9d9b-ed75c2adb449" (UID: "3622d79f-43ba-484c-9d9b-ed75c2adb449"). InnerVolumeSpecName "kube-api-access-7ktj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:42:16.036855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.036849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg" (OuterVolumeSpecName: "kube-api-access-7ghjg") pod "e9f55655-5328-40b7-aaa9-15df2acba799" (UID: "e9f55655-5328-40b7-aaa9-15df2acba799"). InnerVolumeSpecName "kube-api-access-7ghjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:42:16.132617 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.132509    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:16.135557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135537    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c14c00b8-776f-4c5d-908e-a87116339ab4-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135557 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135561    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ghjg\" (UniqueName: \"kubernetes.io/projected/e9f55655-5328-40b7-aaa9-15df2acba799-kube-api-access-7ghjg\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135589    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c14c00b8-776f-4c5d-908e-a87116339ab4-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135604    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3622d79f-43ba-484c-9d9b-ed75c2adb449-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135617    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dbx\" (UniqueName: \"kubernetes.io/projected/c14c00b8-776f-4c5d-908e-a87116339ab4-kube-api-access-h2dbx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135629    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f55655-5328-40b7-aaa9-15df2acba799-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.135682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.135638    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ktj4\" (UniqueName: \"kubernetes.io/projected/3622d79f-43ba-484c-9d9b-ed75c2adb449-kube-api-access-7ktj4\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:42:16.466760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.466676    2583 generic.go:356] "Generic (PLEG): container finished" podID="e9f55655-5328-40b7-aaa9-15df2acba799" containerID="3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950" exitCode=0
Jul 30 21:42:16.466760 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.466724    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"
Jul 30 21:42:16.466966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.466761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" event={"ID":"e9f55655-5328-40b7-aaa9-15df2acba799","Type":"ContainerDied","Data":"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"}
Jul 30 21:42:16.466966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.466820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n" event={"ID":"e9f55655-5328-40b7-aaa9-15df2acba799","Type":"ContainerDied","Data":"ecdffb0ddc853b616962aba13528f1c976f298440196e7126f7f5067eb417a68"}
Jul 30 21:42:16.466966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.466847    2583 scope.go:117] "RemoveContainer" containerID="3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"
Jul 30 21:42:16.468265 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.468240    2583 generic.go:356] "Generic (PLEG): container finished" podID="c14c00b8-776f-4c5d-908e-a87116339ab4" containerID="54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f" exitCode=0
Jul 30 21:42:16.468381 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.468281    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"
Jul 30 21:42:16.468432 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.468379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" event={"ID":"c14c00b8-776f-4c5d-908e-a87116339ab4","Type":"ContainerDied","Data":"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"}
Jul 30 21:42:16.468480 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.468436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r" event={"ID":"c14c00b8-776f-4c5d-908e-a87116339ab4","Type":"ContainerDied","Data":"ceb43eae07f45fb680374b90b819d955417bc106f560168bd995fdd07041c221"}
Jul 30 21:42:16.470238 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.470214    2583 generic.go:356] "Generic (PLEG): container finished" podID="3622d79f-43ba-484c-9d9b-ed75c2adb449" containerID="c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b" exitCode=0
Jul 30 21:42:16.470369 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.470293    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" event={"ID":"3622d79f-43ba-484c-9d9b-ed75c2adb449","Type":"ContainerDied","Data":"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"}
Jul 30 21:42:16.470369 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.470358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr" event={"ID":"3622d79f-43ba-484c-9d9b-ed75c2adb449","Type":"ContainerDied","Data":"bece0d72c4b6ba03c10b394d567b79c5ebf141e81ae088572134b2d268c3577d"}
Jul 30 21:42:16.470465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.470425    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"
Jul 30 21:42:16.477532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.477516    2583 scope.go:117] "RemoveContainer" containerID="3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"
Jul 30 21:42:16.477837 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:42:16.477812    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950\": container with ID starting with 3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950 not found: ID does not exist" containerID="3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"
Jul 30 21:42:16.477919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.477849    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950"} err="failed to get container status \"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950\": rpc error: code = NotFound desc = could not find container \"3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950\": container with ID starting with 3c0356f9293972738e97162d75aac327bdfd439e76be71882bb5629c8f251950 not found: ID does not exist"
Jul 30 21:42:16.477919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.477871    2583 scope.go:117] "RemoveContainer" containerID="54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"
Jul 30 21:42:16.486653 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.486628    2583 scope.go:117] "RemoveContainer" containerID="54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"
Jul 30 21:42:16.486950 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:42:16.486931    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f\": container with ID starting with 54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f not found: ID does not exist" containerID="54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"
Jul 30 21:42:16.487043 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.486960    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f"} err="failed to get container status \"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f\": rpc error: code = NotFound desc = could not find container \"54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f\": container with ID starting with 54adfb3c4fd6677097514d68353b1d8ba5c05ca39dec85ed3abc246b8080949f not found: ID does not exist"
Jul 30 21:42:16.487043 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.486982    2583 scope.go:117] "RemoveContainer" containerID="c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"
Jul 30 21:42:16.494221 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.494196    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:42:16.496365 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.496346    2583 scope.go:117] "RemoveContainer" containerID="c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"
Jul 30 21:42:16.496679 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:42:16.496658    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b\": container with ID starting with c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b not found: ID does not exist" containerID="c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"
Jul 30 21:42:16.496736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.496686    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b"} err="failed to get container status \"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b\": rpc error: code = NotFound desc = could not find container \"c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b\": container with ID starting with c06d67be3bb29644c879e7c5a46779618e6c6a7e11c24cfccae78fdf1f97467b not found: ID does not exist"
Jul 30 21:42:16.497852 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.497834    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d8e5fa-7876b77f56-lzh2r"]
Jul 30 21:42:16.506342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.506320    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:42:16.507723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.507704    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d8e5fa-866f8bdcd4-8hgbr"]
Jul 30 21:42:16.516992 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.516969    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:42:16.518994 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.518974    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d8e5fa-59b98d6c76-kxp8n"]
Jul 30 21:42:16.777009 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.776969    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135e9b21-cf78-4690-afc7-428eef1e8c05" path="/var/lib/kubelet/pods/135e9b21-cf78-4690-afc7-428eef1e8c05/volumes"
Jul 30 21:42:16.777507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.777481    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3622d79f-43ba-484c-9d9b-ed75c2adb449" path="/var/lib/kubelet/pods/3622d79f-43ba-484c-9d9b-ed75c2adb449/volumes"
Jul 30 21:42:16.777925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.777905    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c14c00b8-776f-4c5d-908e-a87116339ab4" path="/var/lib/kubelet/pods/c14c00b8-776f-4c5d-908e-a87116339ab4/volumes"
Jul 30 21:42:16.778267 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:16.778250    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f55655-5328-40b7-aaa9-15df2acba799" path="/var/lib/kubelet/pods/e9f55655-5328-40b7-aaa9-15df2acba799/volumes"
Jul 30 21:42:21.970206 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:21.970151    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:22.033063 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:22.033027    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:22.234779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:22.234738    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:22.469114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:22.469083    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:22.842079 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:22.842035    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:23.121945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:23.121846    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:23.505766 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:23.505735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:25.607878 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:25.607841    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:25.709809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:25.709770    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:26.231058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:26.231013    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:26.355973 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:26.355928    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:27.436903 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.436868    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437190    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c14c00b8-776f-4c5d-908e-a87116339ab4" containerName="maas-api"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437200    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c14c00b8-776f-4c5d-908e-a87116339ab4" containerName="maas-api"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437250    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135e9b21-cf78-4690-afc7-428eef1e8c05" containerName="revoke-keys"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437258    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="135e9b21-cf78-4690-afc7-428eef1e8c05" containerName="revoke-keys"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437270    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f55655-5328-40b7-aaa9-15df2acba799" containerName="payload-processing"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437275    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f55655-5328-40b7-aaa9-15df2acba799" containerName="payload-processing"
Jul 30 21:42:27.437281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437283    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3622d79f-43ba-484c-9d9b-ed75c2adb449" containerName="payload-pre-processing"
Jul 30 21:42:27.437506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437288    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3622d79f-43ba-484c-9d9b-ed75c2adb449" containerName="payload-pre-processing"
Jul 30 21:42:27.437506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437357    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3622d79f-43ba-484c-9d9b-ed75c2adb449" containerName="payload-pre-processing"
Jul 30 21:42:27.437506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437363    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="135e9b21-cf78-4690-afc7-428eef1e8c05" containerName="revoke-keys"
Jul 30 21:42:27.437506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437372    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c14c00b8-776f-4c5d-908e-a87116339ab4" containerName="maas-api"
Jul 30 21:42:27.437506 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.437377    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f55655-5328-40b7-aaa9-15df2acba799" containerName="payload-processing"
Jul 30 21:42:27.442262 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.442243    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.444193 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.444172    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-2abe52-serving-cert\""
Jul 30 21:42:27.444332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.444213    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58scw\""
Jul 30 21:42:27.451714 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.451675    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:42:27.465790 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.465734    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:27.520912 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.520877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2689k\" (UniqueName: \"kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.521093 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.520934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.521093 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.521014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.622417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.622386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2689k\" (UniqueName: \"kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.622417 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.622431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.622698 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.622460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.623111 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.623071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.625039 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.625020    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.630721 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.630696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2689k\" (UniqueName: \"kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k\") pod \"maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.665906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.665871    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:27.753673 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.753634    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:27.899412 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.899379    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:42:27.903087 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:27.903056    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c1718b_86d1_4f41_8ad3_dc8b1a092994.slice/crio-f8ac929702a2f4a3374c9aea200204648b42671322a0399b607870d9d5038f65 WatchSource:0}: Error finding container f8ac929702a2f4a3374c9aea200204648b42671322a0399b607870d9d5038f65: Status 404 returned error can't find the container with id f8ac929702a2f4a3374c9aea200204648b42671322a0399b607870d9d5038f65
Jul 30 21:42:27.928422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:27.928394    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:28.028553 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.028455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:28.082825 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.082793    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:42:28.087439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.087407    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.089379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.089354    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-2abe52\""
Jul 30 21:42:28.089513 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.089492    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-2abe52-dockercfg-jtsqk\""
Jul 30 21:42:28.093384 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.093356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:42:28.105848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.105824    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:42:28.109104 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.109088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.119038 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.119019    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:42:28.127229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.127208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.127352 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.127244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc9sz\" (UniqueName: \"kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.127352 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.127289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.127352 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.127330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgl5h\" (UniqueName: \"kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.228201 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.228141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.228416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.228217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgl5h\" (UniqueName: \"kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.228416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.228291    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.228416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.228333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc9sz\" (UniqueName: \"kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.229166 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.229132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.229294 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.229171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.235936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.235915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgl5h\" (UniqueName: \"kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h\") pod \"payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.236028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.235955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc9sz\" (UniqueName: \"kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz\") pod \"payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.400904 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.400863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:28.421058 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.421033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:28.538685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.538509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" event={"ID":"c8c1718b-86d1-4f41-8ad3-dc8b1a092994","Type":"ContainerStarted","Data":"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"}
Jul 30 21:42:28.538685 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.538631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" event={"ID":"c8c1718b-86d1-4f41-8ad3-dc8b1a092994","Type":"ContainerStarted","Data":"f8ac929702a2f4a3374c9aea200204648b42671322a0399b607870d9d5038f65"}
Jul 30 21:42:28.539621 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.538766    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:28.556858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.556448    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" podStartSLOduration=1.167581841 podStartE2EDuration="1.556431798s" totalImagesPullingTime="388.849957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:28.5554011 +0000 UTC m=+2720.321362984" watchObservedRunningTime="2026-07-30 21:42:28.556431798 +0000 UTC m=+2720.322393690"
Jul 30 21:42:28.593312 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.593189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:42:28.598191 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:28.598162    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda779da23_21f4_4689_a35a_d32bcf97642a.slice/crio-e2e7f3ae21ddf4fb6ecb81a7661c7ff697a303a01bcf9614d747d1ad47723719 WatchSource:0}: Error finding container e2e7f3ae21ddf4fb6ecb81a7661c7ff697a303a01bcf9614d747d1ad47723719: Status 404 returned error can't find the container with id e2e7f3ae21ddf4fb6ecb81a7661c7ff697a303a01bcf9614d747d1ad47723719
Jul 30 21:42:28.617279 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:28.617254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:42:28.619311 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:28.619282    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e718140_09a0_464d_bed5_c569885a92c2.slice/crio-e578033c6d4f74c467873212c452683c534ba1b1a3dd4994945caf013df25479 WatchSource:0}: Error finding container e578033c6d4f74c467873212c452683c534ba1b1a3dd4994945caf013df25479: Status 404 returned error can't find the container with id e578033c6d4f74c467873212c452683c534ba1b1a3dd4994945caf013df25479
Jul 30 21:42:29.544364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.544311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" event={"ID":"4e718140-09a0-464d-bed5-c569885a92c2","Type":"ContainerStarted","Data":"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"}
Jul 30 21:42:29.544814 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.544485    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:29.544814 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.544515    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" event={"ID":"4e718140-09a0-464d-bed5-c569885a92c2","Type":"ContainerStarted","Data":"e578033c6d4f74c467873212c452683c534ba1b1a3dd4994945caf013df25479"}
Jul 30 21:42:29.546003 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.545978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" event={"ID":"a779da23-21f4-4689-a35a-d32bcf97642a","Type":"ContainerStarted","Data":"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"}
Jul 30 21:42:29.546110 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.546023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" event={"ID":"a779da23-21f4-4689-a35a-d32bcf97642a","Type":"ContainerStarted","Data":"e2e7f3ae21ddf4fb6ecb81a7661c7ff697a303a01bcf9614d747d1ad47723719"}
Jul 30 21:42:29.546182 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.546166    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:29.559006 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.558966    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" podStartSLOduration=1.558953293 podStartE2EDuration="1.558953293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:29.557066158 +0000 UTC m=+2721.323028037" watchObservedRunningTime="2026-07-30 21:42:29.558953293 +0000 UTC m=+2721.324915172"
Jul 30 21:42:29.571191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:29.571149    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" podStartSLOduration=1.571134593 podStartE2EDuration="1.571134593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:29.569770226 +0000 UTC m=+2721.335732105" watchObservedRunningTime="2026-07-30 21:42:29.571134593 +0000 UTC m=+2721.337096471"
Jul 30 21:42:34.551917 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:34.551884    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:42:40.553031 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:40.552998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:42:40.555824 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:40.553074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:42:50.699914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:50.699866    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:50.755968 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:50.755927    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:51.228421 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:51.228380    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:51.560255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:51.560127    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:51.871878 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:51.871795    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:52.103681 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:52.103629    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:54.411042 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:54.411000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:54.534100 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:54.534064    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:55.521169 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.521139    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:42:55.524855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.524831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.526537 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.526513    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-592c1e-serving-cert\""
Jul 30 21:42:55.532937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.532913    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:42:55.553351 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.553322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:55.664611 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.664553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9f8d\" (UniqueName: \"kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.664796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.664697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.664796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.664736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.688918 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.688885    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:55.765703 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.765672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.765901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.765709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.765901 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.765737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9f8d\" (UniqueName: \"kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.766379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.766353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.768240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.768214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.772781 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.772721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9f8d\" (UniqueName: \"kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d\") pod \"maas-api-e2e-shared-b-592c1e-67755885c-rnprd\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.836741 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.836706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:55.970629 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:55.970599    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:42:55.973771 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:55.973723    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5849ede3_621f_402e_b41d_7755c5e216c5.slice/crio-a25c96a57263d58d1ef4032f80f9913e06896ee810afefd53f50ae1a9e25484f WatchSource:0}: Error finding container a25c96a57263d58d1ef4032f80f9913e06896ee810afefd53f50ae1a9e25484f: Status 404 returned error can't find the container with id a25c96a57263d58d1ef4032f80f9913e06896ee810afefd53f50ae1a9e25484f
Jul 30 21:42:56.142063 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.141976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:56.245896 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.245867    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:42:56.248542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.248507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:42:56.250418 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.250400    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.252146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.252127    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-592c1e\""
Jul 30 21:42:56.252264 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.252246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-592c1e-dockercfg-j2qnz\""
Jul 30 21:42:56.258864 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.258829    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:42:56.263504 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.263480    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:42:56.267114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.267096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.276065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.276044    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:42:56.372371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.372337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzw49\" (UniqueName: \"kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.372628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.372392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.372628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.372438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.372628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.372481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvdk\" (UniqueName: \"kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.473128 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.473088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzw49\" (UniqueName: \"kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.473299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.473149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.473299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.473178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.473299 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.473211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvdk\" (UniqueName: \"kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.474120 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.474095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.474189 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.474095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.480951 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.480923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvdk\" (UniqueName: \"kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk\") pod \"payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.481052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.480978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzw49\" (UniqueName: \"kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49\") pod \"payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") " pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.563469 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.563433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:56.579056 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.579026    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:56.679523 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.679491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" event={"ID":"5849ede3-621f-402e-b41d-7755c5e216c5","Type":"ContainerStarted","Data":"103a5912cb379189bd8f0e34fe48f9fd117174ce466ab698accf925e89d92c12"}
Jul 30 21:42:56.679689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.679552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" event={"ID":"5849ede3-621f-402e-b41d-7755c5e216c5","Type":"ContainerStarted","Data":"a25c96a57263d58d1ef4032f80f9913e06896ee810afefd53f50ae1a9e25484f"}
Jul 30 21:42:56.679884 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.679852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:42:56.697803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.697741    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" podStartSLOduration=1.279306288 podStartE2EDuration="1.697719173s" totalImagesPullingTime="418.412885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:56.693809602 +0000 UTC m=+2748.459771483" watchObservedRunningTime="2026-07-30 21:42:56.697719173 +0000 UTC m=+2748.463681053"
Jul 30 21:42:56.720526 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.720498    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:42:56.723417 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:56.723382    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1741a4_293b_405d_b9c8_6ceb85391cfe.slice/crio-8add70e4a814b1240c8ab78a129ed8ff5224bb2ccbcb989927fc9dace2a0c196 WatchSource:0}: Error finding container 8add70e4a814b1240c8ab78a129ed8ff5224bb2ccbcb989927fc9dace2a0c196: Status 404 returned error can't find the container with id 8add70e4a814b1240c8ab78a129ed8ff5224bb2ccbcb989927fc9dace2a0c196
Jul 30 21:42:56.747323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:56.747302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:42:56.749670 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:42:56.749631    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76b0713_31b1_4395_88cf_4c73e7f3a951.slice/crio-9ff79b4985e3a93752f87731b7fadf857277a24c1af88d53dd612d93caa5cc44 WatchSource:0}: Error finding container 9ff79b4985e3a93752f87731b7fadf857277a24c1af88d53dd612d93caa5cc44: Status 404 returned error can't find the container with id 9ff79b4985e3a93752f87731b7fadf857277a24c1af88d53dd612d93caa5cc44
Jul 30 21:42:57.686816 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.686759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" event={"ID":"ca1741a4-293b-405d-b9c8-6ceb85391cfe","Type":"ContainerStarted","Data":"d72961df84650eb365381a175d3530dcc0ece50370b46f6806e5a1b371cf3a23"}
Jul 30 21:42:57.687301 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.686841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" event={"ID":"ca1741a4-293b-405d-b9c8-6ceb85391cfe","Type":"ContainerStarted","Data":"8add70e4a814b1240c8ab78a129ed8ff5224bb2ccbcb989927fc9dace2a0c196"}
Jul 30 21:42:57.687301 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.686873    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:42:57.688730 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.688700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" event={"ID":"a76b0713-31b1-4395-88cf-4c73e7f3a951","Type":"ContainerStarted","Data":"32ddb4b57356ec002d9ea3ff80f879ed3e499a94d869be3e15568159dbd685ad"}
Jul 30 21:42:57.688887 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.688753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" event={"ID":"a76b0713-31b1-4395-88cf-4c73e7f3a951","Type":"ContainerStarted","Data":"9ff79b4985e3a93752f87731b7fadf857277a24c1af88d53dd612d93caa5cc44"}
Jul 30 21:42:57.688887 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.688796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:42:57.701040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.700988    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" podStartSLOduration=1.700971809 podStartE2EDuration="1.700971809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:57.70015396 +0000 UTC m=+2749.466115841" watchObservedRunningTime="2026-07-30 21:42:57.700971809 +0000 UTC m=+2749.466933694"
Jul 30 21:42:57.714459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:42:57.714412    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" podStartSLOduration=1.7143986679999998 podStartE2EDuration="1.714398668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:42:57.71254468 +0000 UTC m=+2749.478506559" watchObservedRunningTime="2026-07-30 21:42:57.714398668 +0000 UTC m=+2749.480360586"
Jul 30 21:43:02.694307 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:02.694278    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:43:08.695880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:08.695843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:43:08.696281 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:08.695914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:43:19.387186 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:19.387100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:43:46.432828 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.432789    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:43:46.437899 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.437877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.440059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.440035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"default-dockercfg-s9rlm\""
Jul 30 21:43:46.440204 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.440105    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"kube-root-ca.crt\""
Jul 30 21:43:46.440204 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.440111    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"auth-test-model-2abe52-kserve-self-signed-certs\""
Jul 30 21:43:46.440204 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.440132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"openshift-service-ca.crt\""
Jul 30 21:43:46.449650 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.449602    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:43:46.454719 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.454696    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:43:46.501389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs28p\" (UniqueName: \"kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.501593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.501593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.501593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.501779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.501779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.501651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs28p\" (UniqueName: \"kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602532    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.602919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.603030 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.603030 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.602976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.604855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.604827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.605218 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.605201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.609533 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.609504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs28p\" (UniqueName: \"kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p\") pod \"auth-test-model-2abe52-kserve-6f89d466b7-ncdg9\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") " pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.748680 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.748625    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:46.892123 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.892097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:43:46.897803 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:43:46.897783    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0564500c_497f_4546_96f2_16becd3c32d3.slice/crio-076c2b1965a8e05e74fcf77776e525d51649d726dc08a00dfdc4f5acf660adb2 WatchSource:0}: Error finding container 076c2b1965a8e05e74fcf77776e525d51649d726dc08a00dfdc4f5acf660adb2: Status 404 returned error can't find the container with id 076c2b1965a8e05e74fcf77776e525d51649d726dc08a00dfdc4f5acf660adb2
Jul 30 21:43:46.939060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:46.939032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" event={"ID":"0564500c-497f-4546-96f2-16becd3c32d3","Type":"ContainerStarted","Data":"076c2b1965a8e05e74fcf77776e525d51649d726dc08a00dfdc4f5acf660adb2"}
Jul 30 21:43:47.945836 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:47.945796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" event={"ID":"0564500c-497f-4546-96f2-16becd3c32d3","Type":"ContainerStarted","Data":"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"}
Jul 30 21:43:47.960856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:47.960805    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" podStartSLOduration=1.9607899610000001 podStartE2EDuration="1.960789961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:43:47.959742233 +0000 UTC m=+2799.725704115" watchObservedRunningTime="2026-07-30 21:43:47.960789961 +0000 UTC m=+2799.726751840"
Jul 30 21:43:56.749131 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:56.749078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:56.749663 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:56.749379    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:56.761897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:56.761871    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:43:57.001700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:43:57.001615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:44:08.844479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.844441    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:44:08.848490 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.848466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.850641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.850612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"auth-test-model-592c1e-kserve-self-signed-certs\""
Jul 30 21:44:08.850778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.850649    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"openshift-service-ca.crt\""
Jul 30 21:44:08.850778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.850671    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"kube-root-ca.crt\""
Jul 30 21:44:08.850867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.850777    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"default-dockercfg-l4pch\""
Jul 30 21:44:08.858864 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.858832    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:44:08.870720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.870691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:44:08.894037 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.894339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.894339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.894339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.894339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8pkv\" (UniqueName: \"kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.894339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.894244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.994965 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.994930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.994991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995025    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8pkv\" (UniqueName: \"kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995178 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995431    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.995897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.995781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.997705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.997680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:08.997892 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:08.997874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:09.002485 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:09.002461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8pkv\" (UniqueName: \"kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv\") pod \"auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") " pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:09.160773 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:09.160673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:09.301711 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:09.301685    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:44:09.304190 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:44:09.304158    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0d02f97_4728_4392_a0b9_6451a4e72d34.slice/crio-4fc66afe36a29ea39faa40f605d17679ade7685aacca8e8ebb8f7a17a53770f6 WatchSource:0}: Error finding container 4fc66afe36a29ea39faa40f605d17679ade7685aacca8e8ebb8f7a17a53770f6: Status 404 returned error can't find the container with id 4fc66afe36a29ea39faa40f605d17679ade7685aacca8e8ebb8f7a17a53770f6
Jul 30 21:44:10.058728 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:10.058688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" event={"ID":"f0d02f97-4728-4392-a0b9-6451a4e72d34","Type":"ContainerStarted","Data":"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"}
Jul 30 21:44:10.059138 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:10.058736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" event={"ID":"f0d02f97-4728-4392-a0b9-6451a4e72d34","Type":"ContainerStarted","Data":"4fc66afe36a29ea39faa40f605d17679ade7685aacca8e8ebb8f7a17a53770f6"}
Jul 30 21:44:10.074034 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:10.073971    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" podStartSLOduration=2.073951035 podStartE2EDuration="2.073951035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:44:10.072917118 +0000 UTC m=+2821.838878998" watchObservedRunningTime="2026-07-30 21:44:10.073951035 +0000 UTC m=+2821.839912915"
Jul 30 21:44:19.161049 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:19.161002    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:19.161488 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:19.161066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:19.173778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:19.173740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:20.120474 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:20.120444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:44:40.289164 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:44:40.289101    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:45:00.139815 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.139725    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"]
Jul 30 21:45:00.143524 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.143502    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"
Jul 30 21:45:00.145493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.145472    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:45:00.151720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.151696    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"]
Jul 30 21:45:00.155188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.155169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"
Jul 30 21:45:00.294466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.294441    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"]
Jul 30 21:45:00.297337 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:45:00.297307    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ecd983_798d_41c4_82cf_942f23ba10b4.slice/crio-4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5 WatchSource:0}: Error finding container 4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5: Status 404 returned error can't find the container with id 4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5
Jul 30 21:45:00.316941 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:00.316915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj" event={"ID":"41ecd983-798d-41c4-82cf-942f23ba10b4","Type":"ContainerStarted","Data":"4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5"}
Jul 30 21:45:01.323539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:01.323502    2583 generic.go:356] "Generic (PLEG): container finished" podID="41ecd983-798d-41c4-82cf-942f23ba10b4" containerID="7b388f146913317c88234e1a062c62c521c967101cac84d7acb3046df5325f9e" exitCode=0
Jul 30 21:45:01.323937 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:01.323596    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj" event={"ID":"41ecd983-798d-41c4-82cf-942f23ba10b4","Type":"ContainerDied","Data":"7b388f146913317c88234e1a062c62c521c967101cac84d7acb3046df5325f9e"}
Jul 30 21:45:02.470357 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:02.470326    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"
Jul 30 21:45:03.334795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:03.334755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj" event={"ID":"41ecd983-798d-41c4-82cf-942f23ba10b4","Type":"ContainerDied","Data":"4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5"}
Jul 30 21:45:03.334795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:03.334791    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"
Jul 30 21:45:03.335010 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:45:03.334833    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e0c08f2d1c096e23fef35e6ca2ada3c6adc771c6965c086378065eda1f560e5"
Jul 30 21:46:33.635409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:46:33.635331    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:47:05.419525 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:05.419488    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:47:09.231929 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:09.231901    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:47:09.237215 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:09.237184    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:47:09.413024 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:09.412993    2583 scope.go:117] "RemoveContainer" containerID="3cae86b82b236bad34bb95bcb90e9513966c5c0d66db5c2d7825d5d883aacf28"
Jul 30 21:47:18.864191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:18.864154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:47:18.918118 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:47:18.918079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:48:09.434194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:09.434101    2583 scope.go:117] "RemoveContainer" containerID="60a34cd003c1ce624f9b5f1807fc519d11d884b4524ca97b7c74ec84f6ec011f"
Jul 30 21:48:09.442971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:09.442951    2583 scope.go:117] "RemoveContainer" containerID="1ba150a21cbb6f3aacec2325a45fbb0597d1d415a70ee868a9a02cf53bf23b1a"
Jul 30 21:48:24.784659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:24.784627    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:48:24.806147 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:24.806110    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:48:25.752429 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:25.752393    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:48:25.770930 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:25.770868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:48:39.814332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:39.814289    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" podUID="0564500c-497f-4546-96f2-16becd3c32d3" containerName="main" containerID="cri-o://793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264" gracePeriod=45
Jul 30 21:48:40.078470 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.078409    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:48:40.237139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237095    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237139 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237140    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237180    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fs28p\" (UniqueName: \"kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237311    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home\") pod \"0564500c-497f-4546-96f2-16becd3c32d3\" (UID: \"0564500c-497f-4546-96f2-16becd3c32d3\") "
Jul 30 21:48:40.237477 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237419    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:40.237739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237515    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:40.237739 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.237681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home" (OuterVolumeSpecName: "home") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:40.239780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.239755    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:48:40.239911 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.239856    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm" (OuterVolumeSpecName: "dshm") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:40.239911 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.239858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p" (OuterVolumeSpecName: "kube-api-access-fs28p") pod "0564500c-497f-4546-96f2-16becd3c32d3" (UID: "0564500c-497f-4546-96f2-16becd3c32d3"). InnerVolumeSpecName "kube-api-access-fs28p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338602    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338636    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338644    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0564500c-497f-4546-96f2-16becd3c32d3-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338652    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs28p\" (UniqueName: \"kubernetes.io/projected/0564500c-497f-4546-96f2-16becd3c32d3-kube-api-access-fs28p\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338660    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.338693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.338670    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0564500c-497f-4546-96f2-16becd3c32d3-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:40.468447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.468409    2583 generic.go:356] "Generic (PLEG): container finished" podID="0564500c-497f-4546-96f2-16becd3c32d3" containerID="793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264" exitCode=0
Jul 30 21:48:40.468660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.468455    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"
Jul 30 21:48:40.468660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.468495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" event={"ID":"0564500c-497f-4546-96f2-16becd3c32d3","Type":"ContainerDied","Data":"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"}
Jul 30 21:48:40.468660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.468544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9" event={"ID":"0564500c-497f-4546-96f2-16becd3c32d3","Type":"ContainerDied","Data":"076c2b1965a8e05e74fcf77776e525d51649d726dc08a00dfdc4f5acf660adb2"}
Jul 30 21:48:40.468660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.468592    2583 scope.go:117] "RemoveContainer" containerID="793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"
Jul 30 21:48:40.477793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.477776    2583 scope.go:117] "RemoveContainer" containerID="793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"
Jul 30 21:48:40.478071 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:48:40.478046    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264\": container with ID starting with 793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264 not found: ID does not exist" containerID="793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"
Jul 30 21:48:40.478179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.478076    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264"} err="failed to get container status \"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264\": rpc error: code = NotFound desc = could not find container \"793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264\": container with ID starting with 793b191e0344997eed96a0ebfacb41b791062dfab0ac0a2b375078cf4c814264 not found: ID does not exist"
Jul 30 21:48:40.488853 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.488823    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:48:40.494652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.494624    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/auth-test-model-2abe52-kserve-6f89d466b7-ncdg9"]
Jul 30 21:48:40.772381 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.772333    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" podUID="f0d02f97-4728-4392-a0b9-6451a4e72d34" containerName="main" containerID="cri-o://5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f" gracePeriod=45
Jul 30 21:48:40.777996 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:40.777969    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0564500c-497f-4546-96f2-16becd3c32d3" path="/var/lib/kubelet/pods/0564500c-497f-4546-96f2-16becd3c32d3/volumes"
Jul 30 21:48:41.029039 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.028983    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:48:41.146181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146150    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146256    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146273    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8pkv\" (UniqueName: \"kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home\") pod \"f0d02f97-4728-4392-a0b9-6451a4e72d34\" (UID: \"f0d02f97-4728-4392-a0b9-6451a4e72d34\") "
Jul 30 21:48:41.146689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146457    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:41.146689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:41.146689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146558    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.146689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146628    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.146689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.146676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home" (OuterVolumeSpecName: "home") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:41.148757 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.148733    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv" (OuterVolumeSpecName: "kube-api-access-r8pkv") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "kube-api-access-r8pkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:48:41.148757 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.148740    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm" (OuterVolumeSpecName: "dshm") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:48:41.148880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.148816    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0d02f97-4728-4392-a0b9-6451a4e72d34" (UID: "f0d02f97-4728-4392-a0b9-6451a4e72d34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:48:41.247787 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.247748    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.247787 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.247781    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8pkv\" (UniqueName: \"kubernetes.io/projected/f0d02f97-4728-4392-a0b9-6451a4e72d34-kube-api-access-r8pkv\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.247787 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.247796    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0d02f97-4728-4392-a0b9-6451a4e72d34-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.248023 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.247807    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0d02f97-4728-4392-a0b9-6451a4e72d34-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:48:41.465690 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.465601    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:48:41.466088 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466068    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d02f97-4728-4392-a0b9-6451a4e72d34" containerName="main"
Jul 30 21:48:41.466179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466091    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d02f97-4728-4392-a0b9-6451a4e72d34" containerName="main"
Jul 30 21:48:41.466179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466103    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ecd983-798d-41c4-82cf-942f23ba10b4" containerName="cleanup"
Jul 30 21:48:41.466179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466110    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ecd983-798d-41c4-82cf-942f23ba10b4" containerName="cleanup"
Jul 30 21:48:41.466179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466124    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0564500c-497f-4546-96f2-16becd3c32d3" containerName="main"
Jul 30 21:48:41.466179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466132    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0564500c-497f-4546-96f2-16becd3c32d3" containerName="main"
Jul 30 21:48:41.466430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466255    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d02f97-4728-4392-a0b9-6451a4e72d34" containerName="main"
Jul 30 21:48:41.466430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466269    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0564500c-497f-4546-96f2-16becd3c32d3" containerName="main"
Jul 30 21:48:41.466430 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.466278    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ecd983-798d-41c4-82cf-942f23ba10b4" containerName="cleanup"
Jul 30 21:48:41.469718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.469694    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.471781 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.471753    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"sub-test-model-2abe52-kserve-self-signed-certs\""
Jul 30 21:48:41.471918 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.471754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"default-dockercfg-s9rlm\""
Jul 30 21:48:41.471918 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.471799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"openshift-service-ca.crt\""
Jul 30 21:48:41.471918 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.471823    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2abe52\"/\"kube-root-ca.crt\""
Jul 30 21:48:41.479017 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.478987    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0d02f97-4728-4392-a0b9-6451a4e72d34" containerID="5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f" exitCode=0
Jul 30 21:48:41.479161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.479040    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"
Jul 30 21:48:41.479161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.479071    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" event={"ID":"f0d02f97-4728-4392-a0b9-6451a4e72d34","Type":"ContainerDied","Data":"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"}
Jul 30 21:48:41.479161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.479121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz" event={"ID":"f0d02f97-4728-4392-a0b9-6451a4e72d34","Type":"ContainerDied","Data":"4fc66afe36a29ea39faa40f605d17679ade7685aacca8e8ebb8f7a17a53770f6"}
Jul 30 21:48:41.479161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.479149    2583 scope.go:117] "RemoveContainer" containerID="5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"
Jul 30 21:48:41.479722 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.479700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:48:41.482212 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.482176    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:48:41.490554 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.490529    2583 scope.go:117] "RemoveContainer" containerID="5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"
Jul 30 21:48:41.490877 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:48:41.490855    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f\": container with ID starting with 5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f not found: ID does not exist" containerID="5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"
Jul 30 21:48:41.490966 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.490887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f"} err="failed to get container status \"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f\": rpc error: code = NotFound desc = could not find container \"5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f\": container with ID starting with 5675a00e9df2f2b38b4099f630a92d6b9f881d4d9af0f336a4d012f6e05db47f not found: ID does not exist"
Jul 30 21:48:41.502334 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.502311    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:48:41.510041 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.510017    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/auth-test-model-592c1e-kserve-6987cbf6ff-lnbmz"]
Jul 30 21:48:41.549980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.549945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.550146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.549988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.550146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.550068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.550146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.550103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvm58\" (UniqueName: \"kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.550146 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.550130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.550289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.550150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.650976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.650940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.650981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvm58\" (UniqueName: \"kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651082    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.651544 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.651514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.653539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.653515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.653759 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.653741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.658749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.658730    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvm58\" (UniqueName: \"kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58\") pod \"sub-test-model-2abe52-kserve-7d859bd969-nnrll\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") " pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.781342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.781316    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:41.923402 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:41.923375    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:48:41.925356 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:48:41.925319    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4934c485_7d00_47a1_b532_a39321de906e.slice/crio-e2b03dba012705ed62a64c9df5152cedf203a7bca5e4e076e8c92ce319a42a7c WatchSource:0}: Error finding container e2b03dba012705ed62a64c9df5152cedf203a7bca5e4e076e8c92ce319a42a7c: Status 404 returned error can't find the container with id e2b03dba012705ed62a64c9df5152cedf203a7bca5e4e076e8c92ce319a42a7c
Jul 30 21:48:42.488064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:42.488029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" event={"ID":"4934c485-7d00-47a1-b532-a39321de906e","Type":"ContainerStarted","Data":"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"}
Jul 30 21:48:42.488551 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:42.488076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" event={"ID":"4934c485-7d00-47a1-b532-a39321de906e","Type":"ContainerStarted","Data":"e2b03dba012705ed62a64c9df5152cedf203a7bca5e4e076e8c92ce319a42a7c"}
Jul 30 21:48:42.510587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:42.507381    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" podStartSLOduration=1.5073607679999999 podStartE2EDuration="1.507360768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:48:42.503594386 +0000 UTC m=+3094.269556258" watchObservedRunningTime="2026-07-30 21:48:42.507360768 +0000 UTC m=+3094.273322649"
Jul 30 21:48:42.779147 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:42.779051    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d02f97-4728-4392-a0b9-6451a4e72d34" path="/var/lib/kubelet/pods/f0d02f97-4728-4392-a0b9-6451a4e72d34/volumes"
Jul 30 21:48:51.782241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:51.782191    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:51.782241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:51.782254    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:51.798652 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:51.798631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:48:52.549076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:48:52.549042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:49:12.251698 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.251661    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:49:12.256447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.256425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.258403 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.258379    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"openshift-service-ca.crt\""
Jul 30 21:49:12.258536 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.258416    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"kube-root-ca.crt\""
Jul 30 21:49:12.258746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.258728    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"default-dockercfg-l4pch\""
Jul 30 21:49:12.258829 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.258776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"sub-test-model-592c1e-kserve-self-signed-certs\""
Jul 30 21:49:12.262473 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.262448    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:49:12.273418 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.273384    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:49:12.317678 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.317678 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.317970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.317970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.317970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.317970 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.317838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj5k\" (UniqueName: \"kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418366 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418366 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418658 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418658 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418406    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj5k\" (UniqueName: \"kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418658 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418658 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418535    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.418990 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.418962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.419051 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.419007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.420891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.420861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.421134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.421116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.426501 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.426478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj5k\" (UniqueName: \"kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k\") pod \"sub-test-model-592c1e-kserve-789667f58f-5mm54\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") " pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.566922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.566832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:12.714297 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:12.714271    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:49:12.716353 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:49:12.716323    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349fa78b_0c2f_41d5_8acc_da670be7e536.slice/crio-136fff6c68ec6ddd1b81ef8496a1716f696f3604175f16748e9290903a96273e WatchSource:0}: Error finding container 136fff6c68ec6ddd1b81ef8496a1716f696f3604175f16748e9290903a96273e: Status 404 returned error can't find the container with id 136fff6c68ec6ddd1b81ef8496a1716f696f3604175f16748e9290903a96273e
Jul 30 21:49:13.643817 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:13.643778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" event={"ID":"349fa78b-0c2f-41d5-8acc-da670be7e536","Type":"ContainerStarted","Data":"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"}
Jul 30 21:49:13.643817 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:13.643817    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" event={"ID":"349fa78b-0c2f-41d5-8acc-da670be7e536","Type":"ContainerStarted","Data":"136fff6c68ec6ddd1b81ef8496a1716f696f3604175f16748e9290903a96273e"}
Jul 30 21:49:13.658656 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:13.658605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" podStartSLOduration=1.65859039 podStartE2EDuration="1.65859039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:49:13.657196637 +0000 UTC m=+3125.423158526" watchObservedRunningTime="2026-07-30 21:49:13.65859039 +0000 UTC m=+3125.424552263"
Jul 30 21:49:22.567775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:22.567740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:22.567775 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:22.567786    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:22.580493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:22.580468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:49:22.707216 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:49:22.707178    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:50:36.735436 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:36.735397    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:50:36.755832 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:36.755794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:50:37.718546 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:37.718511    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:50:37.736807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:37.736775    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:50:47.969646 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:47.969603    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:50:51.755499 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:51.755441    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" podUID="4934c485-7d00-47a1-b532-a39321de906e" containerName="main" containerID="cri-o://ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a" gracePeriod=45
Jul 30 21:50:52.011530 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.011472    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:50:52.104400 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104361    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104411    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104445    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104492    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvm58\" (UniqueName: \"kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104616    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache\") pod \"4934c485-7d00-47a1-b532-a39321de906e\" (UID: \"4934c485-7d00-47a1-b532-a39321de906e\") "
Jul 30 21:50:52.104905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104748    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:52.104905 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home" (OuterVolumeSpecName: "home") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:52.105015 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104942    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.105015 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104964    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.105015 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.104971    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:52.106952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.106917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58" (OuterVolumeSpecName: "kube-api-access-lvm58") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "kube-api-access-lvm58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:50:52.107120 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.107101    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm" (OuterVolumeSpecName: "dshm") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:52.107182 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.107137    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4934c485-7d00-47a1-b532-a39321de906e" (UID: "4934c485-7d00-47a1-b532-a39321de906e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:50:52.152980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.152947    2583 generic.go:356] "Generic (PLEG): container finished" podID="4934c485-7d00-47a1-b532-a39321de906e" containerID="ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a" exitCode=0
Jul 30 21:50:52.153126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.152991    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"
Jul 30 21:50:52.153126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.152995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" event={"ID":"4934c485-7d00-47a1-b532-a39321de906e","Type":"ContainerDied","Data":"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"}
Jul 30 21:50:52.153126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.153029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll" event={"ID":"4934c485-7d00-47a1-b532-a39321de906e","Type":"ContainerDied","Data":"e2b03dba012705ed62a64c9df5152cedf203a7bca5e4e076e8c92ce319a42a7c"}
Jul 30 21:50:52.153126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.153046    2583 scope.go:117] "RemoveContainer" containerID="ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"
Jul 30 21:50:52.162439 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.162424    2583 scope.go:117] "RemoveContainer" containerID="ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"
Jul 30 21:50:52.162697 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:50:52.162678    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a\": container with ID starting with ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a not found: ID does not exist" containerID="ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"
Jul 30 21:50:52.162753 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.162706    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a"} err="failed to get container status \"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a\": rpc error: code = NotFound desc = could not find container \"ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a\": container with ID starting with ee6682a6363c41832327cbb30eefefc7bd223712448a6d4c362f050e8ca3f56a not found: ID does not exist"
Jul 30 21:50:52.172274 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.172248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:50:52.175469 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.175451    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2abe52/sub-test-model-2abe52-kserve-7d859bd969-nnrll"]
Jul 30 21:50:52.206298 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.206269    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvm58\" (UniqueName: \"kubernetes.io/projected/4934c485-7d00-47a1-b532-a39321de906e-kube-api-access-lvm58\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.206298 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.206298    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.206438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.206310    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4934c485-7d00-47a1-b532-a39321de906e-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.206438 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.206318    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4934c485-7d00-47a1-b532-a39321de906e-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:52.738734 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.738694    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" podUID="349fa78b-0c2f-41d5-8acc-da670be7e536" containerName="main" containerID="cri-o://ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941" gracePeriod=45
Jul 30 21:50:52.777153 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.777127    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4934c485-7d00-47a1-b532-a39321de906e" path="/var/lib/kubelet/pods/4934c485-7d00-47a1-b532-a39321de906e/volumes"
Jul 30 21:50:52.994318 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:52.994255    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:50:53.112526 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blj5k\" (UniqueName: \"kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.112748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.112748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.112748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.112748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112708    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.112748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112746    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm\") pod \"349fa78b-0c2f-41d5-8acc-da670be7e536\" (UID: \"349fa78b-0c2f-41d5-8acc-da670be7e536\") "
Jul 30 21:50:53.113018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home" (OuterVolumeSpecName: "home") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:53.113018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:53.113018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.112930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache" (OuterVolumeSpecName: "model-cache") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:53.113018 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.113013    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:53.113179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.113032    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:53.113179 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.113047    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:53.115026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.114992    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k" (OuterVolumeSpecName: "kube-api-access-blj5k") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "kube-api-access-blj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:50:53.115165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.115046    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:50:53.115165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.115143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm" (OuterVolumeSpecName: "dshm") pod "349fa78b-0c2f-41d5-8acc-da670be7e536" (UID: "349fa78b-0c2f-41d5-8acc-da670be7e536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:50:53.161681 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.161634    2583 generic.go:356] "Generic (PLEG): container finished" podID="349fa78b-0c2f-41d5-8acc-da670be7e536" containerID="ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941" exitCode=0
Jul 30 21:50:53.161858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.161683    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"
Jul 30 21:50:53.161858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.161761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" event={"ID":"349fa78b-0c2f-41d5-8acc-da670be7e536","Type":"ContainerDied","Data":"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"}
Jul 30 21:50:53.161858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.161803    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54" event={"ID":"349fa78b-0c2f-41d5-8acc-da670be7e536","Type":"ContainerDied","Data":"136fff6c68ec6ddd1b81ef8496a1716f696f3604175f16748e9290903a96273e"}
Jul 30 21:50:53.161858 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.161824    2583 scope.go:117] "RemoveContainer" containerID="ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"
Jul 30 21:50:53.175644 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.175512    2583 scope.go:117] "RemoveContainer" containerID="ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"
Jul 30 21:50:53.176110 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:50:53.176077    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941\": container with ID starting with ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941 not found: ID does not exist" containerID="ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"
Jul 30 21:50:53.176267 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.176121    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941"} err="failed to get container status \"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941\": rpc error: code = NotFound desc = could not find container \"ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941\": container with ID starting with ccd9644fa68c9ed8c5872af86e4fb5e0bb84fc40fc8223bfd8a14decf266d941 not found: ID does not exist"
Jul 30 21:50:53.187976 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.187952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:50:53.191575 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.191539    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/sub-test-model-592c1e-kserve-789667f58f-5mm54"]
Jul 30 21:50:53.214247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.214223    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj5k\" (UniqueName: \"kubernetes.io/projected/349fa78b-0c2f-41d5-8acc-da670be7e536-kube-api-access-blj5k\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:53.214247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.214247    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349fa78b-0c2f-41d5-8acc-da670be7e536-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:53.214422 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:53.214259    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349fa78b-0c2f-41d5-8acc-da670be7e536-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:50:54.777946 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:50:54.777899    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349fa78b-0c2f-41d5-8acc-da670be7e536" path="/var/lib/kubelet/pods/349fa78b-0c2f-41d5-8acc-da670be7e536/volumes"
Jul 30 21:51:05.338483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.338392    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:51:05.338969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.338861    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4934c485-7d00-47a1-b532-a39321de906e" containerName="main"
Jul 30 21:51:05.338969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.338880    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4934c485-7d00-47a1-b532-a39321de906e" containerName="main"
Jul 30 21:51:05.338969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.338898    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349fa78b-0c2f-41d5-8acc-da670be7e536" containerName="main"
Jul 30 21:51:05.338969 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.338907    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="349fa78b-0c2f-41d5-8acc-da670be7e536" containerName="main"
Jul 30 21:51:05.339232 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.339021    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="349fa78b-0c2f-41d5-8acc-da670be7e536" containerName="main"
Jul 30 21:51:05.339232 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.339035    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4934c485-7d00-47a1-b532-a39321de906e" containerName="main"
Jul 30 21:51:05.343936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.343909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.345939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.345911    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"openshift-service-ca.crt\""
Jul 30 21:51:05.346070 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.345924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"kube-root-ca.crt\""
Jul 30 21:51:05.346358 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.346334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"rate-test-model-592c1e-kserve-self-signed-certs\""
Jul 30 21:51:05.346459 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.346359    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-592c1e\"/\"default-dockercfg-l4pch\""
Jul 30 21:51:05.352480 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.351733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:51:05.398906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.398864    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:51:05.417344 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.417555 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.417555 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dknp\" (UniqueName: \"kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.417555 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.417779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417629    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.417779 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.417667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.518720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.518720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.518993 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.518993 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.518993 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dknp\" (UniqueName: \"kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.519167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.518993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.519167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.519059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.519167 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.519139    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.519326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.519297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.521160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.521144    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.521673 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.521654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.526828 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.526806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dknp\" (UniqueName: \"kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp\") pod \"rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") " pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.655743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.655649    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:05.797880 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:05.797780    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:51:05.800114 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:51:05.800089    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42b29916_e90a_42ff_a945_87a2b315d5c2.slice/crio-034f761c42ccc097913a3d1ec6d2435e57f4d41f71c41d5584de46e642bca5ec WatchSource:0}: Error finding container 034f761c42ccc097913a3d1ec6d2435e57f4d41f71c41d5584de46e642bca5ec: Status 404 returned error can't find the container with id 034f761c42ccc097913a3d1ec6d2435e57f4d41f71c41d5584de46e642bca5ec
Jul 30 21:51:06.237052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:06.237008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" event={"ID":"42b29916-e90a-42ff-a945-87a2b315d5c2","Type":"ContainerStarted","Data":"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"}
Jul 30 21:51:06.237052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:06.237055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" event={"ID":"42b29916-e90a-42ff-a945-87a2b315d5c2","Type":"ContainerStarted","Data":"034f761c42ccc097913a3d1ec6d2435e57f4d41f71c41d5584de46e642bca5ec"}
Jul 30 21:51:06.255155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:06.255097    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" podStartSLOduration=1.255083179 podStartE2EDuration="1.255083179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:51:06.252931181 +0000 UTC m=+3238.018893060" watchObservedRunningTime="2026-07-30 21:51:06.255083179 +0000 UTC m=+3238.021045057"
Jul 30 21:51:15.655944 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:15.655908    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:15.656376 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:15.656037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:15.668706 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:15.668672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:16.297374 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:16.297347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:51:26.185066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:51:26.185022    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:09.260520 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:09.260490    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:52:09.266332 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:09.266311    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:52:14.336925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:14.336890    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:15.243954 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:15.243906    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:52:15.440440 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:15.440392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:16.146998 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.146962    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299523120"]
Jul 30 21:52:16.150465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.150446    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:16.152059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.152039    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rjdr2\""
Jul 30 21:52:16.156171 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.156147    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299523120"]
Jul 30 21:52:16.211331 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.211291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnxn\" (UniqueName: \"kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn\") pod \"test-curl-7240-139825299523120\" (UID: \"603fbf6f-a629-42b1-baa5-1b30ff62748d\") " pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:16.312082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.312039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnxn\" (UniqueName: \"kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn\") pod \"test-curl-7240-139825299523120\" (UID: \"603fbf6f-a629-42b1-baa5-1b30ff62748d\") " pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:16.319407 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.319372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnxn\" (UniqueName: \"kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn\") pod \"test-curl-7240-139825299523120\" (UID: \"603fbf6f-a629-42b1-baa5-1b30ff62748d\") " pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:16.460416 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.460334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:16.606023 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.605992    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299523120"]
Jul 30 21:52:16.608760 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:16.608728    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod603fbf6f_a629_42b1_baa5_1b30ff62748d.slice/crio-9dfb2171c6b442fd7a89cb44af173abf662f9ba493e08610a7b3008f6126a940 WatchSource:0}: Error finding container 9dfb2171c6b442fd7a89cb44af173abf662f9ba493e08610a7b3008f6126a940: Status 404 returned error can't find the container with id 9dfb2171c6b442fd7a89cb44af173abf662f9ba493e08610a7b3008f6126a940
Jul 30 21:52:16.610412 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:16.610390    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:52:17.600492 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:17.600402    2583 generic.go:356] "Generic (PLEG): container finished" podID="603fbf6f-a629-42b1-baa5-1b30ff62748d" containerID="8e9d421228d557c8988e4bec2813d5d613bd068134a71ed361748ef46d9ca43f" exitCode=0
Jul 30 21:52:17.600883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:17.600495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299523120" event={"ID":"603fbf6f-a629-42b1-baa5-1b30ff62748d","Type":"ContainerDied","Data":"8e9d421228d557c8988e4bec2813d5d613bd068134a71ed361748ef46d9ca43f"}
Jul 30 21:52:17.600883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:17.600531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299523120" event={"ID":"603fbf6f-a629-42b1-baa5-1b30ff62748d","Type":"ContainerStarted","Data":"9dfb2171c6b442fd7a89cb44af173abf662f9ba493e08610a7b3008f6126a940"}
Jul 30 21:52:18.751552 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.751524    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:18.769956 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.769928    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299523120_603fbf6f-a629-42b1-baa5-1b30ff62748d/test-curl-7240-139825299523120/0.log"
Jul 30 21:52:18.791345 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.791314    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299523120"]
Jul 30 21:52:18.795286 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.795264    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299523120"]
Jul 30 21:52:18.832046 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.832015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnxn\" (UniqueName: \"kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn\") pod \"603fbf6f-a629-42b1-baa5-1b30ff62748d\" (UID: \"603fbf6f-a629-42b1-baa5-1b30ff62748d\") "
Jul 30 21:52:18.834280 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.834249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn" (OuterVolumeSpecName: "kube-api-access-jrnxn") pod "603fbf6f-a629-42b1-baa5-1b30ff62748d" (UID: "603fbf6f-a629-42b1-baa5-1b30ff62748d"). InnerVolumeSpecName "kube-api-access-jrnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:18.909920 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.909843    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299387568"]
Jul 30 21:52:18.910246 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.910233    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" containerName="test-curl-7240-139825299523120"
Jul 30 21:52:18.910289 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.910247    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" containerName="test-curl-7240-139825299523120"
Jul 30 21:52:18.910324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.910308    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" containerName="test-curl-7240-139825299523120"
Jul 30 21:52:18.913134 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.913118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:18.919126 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.919098    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299387568"]
Jul 30 21:52:18.933693 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:18.933667    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnxn\" (UniqueName: \"kubernetes.io/projected/603fbf6f-a629-42b1-baa5-1b30ff62748d-kube-api-access-jrnxn\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:19.034060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.034026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dgj\" (UniqueName: \"kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj\") pod \"test-curl-7240-139825299387568\" (UID: \"85f4364b-a937-440e-93ff-f4cf8f408cf0\") " pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:19.135338 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.135296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dgj\" (UniqueName: \"kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj\") pod \"test-curl-7240-139825299387568\" (UID: \"85f4364b-a937-440e-93ff-f4cf8f408cf0\") " pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:19.144757 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.144735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dgj\" (UniqueName: \"kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj\") pod \"test-curl-7240-139825299387568\" (UID: \"85f4364b-a937-440e-93ff-f4cf8f408cf0\") " pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:19.224809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.224778    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:19.357909 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.357880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299387568"]
Jul 30 21:52:19.360254 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:19.360226    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85f4364b_a937_440e_93ff_f4cf8f408cf0.slice/crio-27a2dcbd262a79e8fc63d6afd03299fa91c33658c11f633bbf6eed98eeb63137 WatchSource:0}: Error finding container 27a2dcbd262a79e8fc63d6afd03299fa91c33658c11f633bbf6eed98eeb63137: Status 404 returned error can't find the container with id 27a2dcbd262a79e8fc63d6afd03299fa91c33658c11f633bbf6eed98eeb63137
Jul 30 21:52:19.613161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.613080    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:19.613161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.613120    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dfb2171c6b442fd7a89cb44af173abf662f9ba493e08610a7b3008f6126a940"
Jul 30 21:52:19.614240 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:19.614215    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299387568" event={"ID":"85f4364b-a937-440e-93ff-f4cf8f408cf0","Type":"ContainerStarted","Data":"27a2dcbd262a79e8fc63d6afd03299fa91c33658c11f633bbf6eed98eeb63137"}
Jul 30 21:52:19.623217 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:19.623186    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299523120\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:20.626374 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:20.626340    2583 generic.go:356] "Generic (PLEG): container finished" podID="85f4364b-a937-440e-93ff-f4cf8f408cf0" containerID="c175f7f14d40be091cd140e8164e856e51ea5e10f0f4b9665ba1a1c282dbca74" exitCode=0
Jul 30 21:52:20.626796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:20.626424    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299387568" event={"ID":"85f4364b-a937-440e-93ff-f4cf8f408cf0","Type":"ContainerDied","Data":"c175f7f14d40be091cd140e8164e856e51ea5e10f0f4b9665ba1a1c282dbca74"}
Jul 30 21:52:20.640116 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:20.640075    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299523120\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" pod="opendatahub/test-curl-7240-139825299523120"
Jul 30 21:52:20.776872 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:20.776836    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603fbf6f-a629-42b1-baa5-1b30ff62748d" path="/var/lib/kubelet/pods/603fbf6f-a629-42b1-baa5-1b30ff62748d/volumes"
Jul 30 21:52:21.776065 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.776042    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:21.793705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.793623    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299387568_85f4364b-a937-440e-93ff-f4cf8f408cf0/test-curl-7240-139825299387568/0.log"
Jul 30 21:52:21.815110 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.815073    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299387568"]
Jul 30 21:52:21.818129 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.818107    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299387568"]
Jul 30 21:52:21.859827 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.859790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5dgj\" (UniqueName: \"kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj\") pod \"85f4364b-a937-440e-93ff-f4cf8f408cf0\" (UID: \"85f4364b-a937-440e-93ff-f4cf8f408cf0\") "
Jul 30 21:52:21.861941 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.861905    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj" (OuterVolumeSpecName: "kube-api-access-z5dgj") pod "85f4364b-a937-440e-93ff-f4cf8f408cf0" (UID: "85f4364b-a937-440e-93ff-f4cf8f408cf0"). InnerVolumeSpecName "kube-api-access-z5dgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:21.960956 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.960872    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dgj\" (UniqueName: \"kubernetes.io/projected/85f4364b-a937-440e-93ff-f4cf8f408cf0-kube-api-access-z5dgj\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:21.984761 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.984722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:21.985132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.985118    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f4364b-a937-440e-93ff-f4cf8f408cf0" containerName="test-curl-7240-139825299387568"
Jul 30 21:52:21.985132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.985133    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f4364b-a937-440e-93ff-f4cf8f408cf0" containerName="test-curl-7240-139825299387568"
Jul 30 21:52:21.985252 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.985195    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f4364b-a937-440e-93ff-f4cf8f408cf0" containerName="test-curl-7240-139825299387568"
Jul 30 21:52:21.988165 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.988145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:21.993743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:21.993720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:22.061981 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.061942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchx6\" (UniqueName: \"kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6\") pod \"test-curl-7240-139825299451024\" (UID: \"8e4983f7-c593-4547-bef6-afe403d8a66d\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:22.162661 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.162611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchx6\" (UniqueName: \"kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6\") pod \"test-curl-7240-139825299451024\" (UID: \"8e4983f7-c593-4547-bef6-afe403d8a66d\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:22.169701 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.169676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchx6\" (UniqueName: \"kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6\") pod \"test-curl-7240-139825299451024\" (UID: \"8e4983f7-c593-4547-bef6-afe403d8a66d\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:22.299699 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.299666    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:22.432757 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.432733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:22.435316 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:22.435292    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e4983f7_c593_4547_bef6_afe403d8a66d.slice/crio-6b0d3803a0240ab6d48420082c32975e309f10be1bb85520caf46a66aa3edcdc WatchSource:0}: Error finding container 6b0d3803a0240ab6d48420082c32975e309f10be1bb85520caf46a66aa3edcdc: Status 404 returned error can't find the container with id 6b0d3803a0240ab6d48420082c32975e309f10be1bb85520caf46a66aa3edcdc
Jul 30 21:52:22.638783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.638691    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27a2dcbd262a79e8fc63d6afd03299fa91c33658c11f633bbf6eed98eeb63137"
Jul 30 21:52:22.638783 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.638700    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:22.640071 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.640036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299451024" event={"ID":"8e4983f7-c593-4547-bef6-afe403d8a66d","Type":"ContainerStarted","Data":"6b0d3803a0240ab6d48420082c32975e309f10be1bb85520caf46a66aa3edcdc"}
Jul 30 21:52:22.649252 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:22.649227    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299387568\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="85f4364b-a937-440e-93ff-f4cf8f408cf0" pod="opendatahub/test-curl-7240-139825299387568"
Jul 30 21:52:22.777371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:22.777335    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f4364b-a937-440e-93ff-f4cf8f408cf0" path="/var/lib/kubelet/pods/85f4364b-a937-440e-93ff-f4cf8f408cf0/volumes"
Jul 30 21:52:23.647105 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:23.647073    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e4983f7-c593-4547-bef6-afe403d8a66d" containerID="6a64e5f32050e4a0e5f6353a891419cb6bae84e4a8e358b65a5c6c1c3fa04f60" exitCode=0
Jul 30 21:52:23.647279 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:23.647135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299451024" event={"ID":"8e4983f7-c593-4547-bef6-afe403d8a66d","Type":"ContainerDied","Data":"6a64e5f32050e4a0e5f6353a891419cb6bae84e4a8e358b65a5c6c1c3fa04f60"}
Jul 30 21:52:24.823412 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.823379    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:24.839874 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.839843    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299451024_8e4983f7-c593-4547-bef6-afe403d8a66d/test-curl-7240-139825299451024/0.log"
Jul 30 21:52:24.861497 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.861460    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:24.865162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.865136    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:24.885394 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.885374    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gchx6\" (UniqueName: \"kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6\") pod \"8e4983f7-c593-4547-bef6-afe403d8a66d\" (UID: \"8e4983f7-c593-4547-bef6-afe403d8a66d\") "
Jul 30 21:52:24.887591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.887552    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6" (OuterVolumeSpecName: "kube-api-access-gchx6") pod "8e4983f7-c593-4547-bef6-afe403d8a66d" (UID: "8e4983f7-c593-4547-bef6-afe403d8a66d"). InnerVolumeSpecName "kube-api-access-gchx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:24.981463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.981430    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:24.981871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.981854    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e4983f7-c593-4547-bef6-afe403d8a66d" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:24.981871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.981871    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e4983f7-c593-4547-bef6-afe403d8a66d" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:24.982015 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.981935    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e4983f7-c593-4547-bef6-afe403d8a66d" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:24.984866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.984846    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:24.985980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.985956    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchx6\" (UniqueName: \"kubernetes.io/projected/8e4983f7-c593-4547-bef6-afe403d8a66d-kube-api-access-gchx6\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:24.991001 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:24.990977    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:25.086759 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.086720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t\") pod \"test-curl-7240-139825299451024\" (UID: \"10cdc8f9-34a7-4cb6-8522-c9120db1becf\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:25.187217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.187180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t\") pod \"test-curl-7240-139825299451024\" (UID: \"10cdc8f9-34a7-4cb6-8522-c9120db1becf\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:25.194796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.194763    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t\") pod \"test-curl-7240-139825299451024\" (UID: \"10cdc8f9-34a7-4cb6-8522-c9120db1becf\") " pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:25.296700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.296613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:25.445339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.445302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:25.448417 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:25.448373    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10cdc8f9_34a7_4cb6_8522_c9120db1becf.slice/crio-ddc345259f07efb7343fb36cceb4782fc946256559fd8f48cbc0d31bb9c914c4 WatchSource:0}: Error finding container ddc345259f07efb7343fb36cceb4782fc946256559fd8f48cbc0d31bb9c914c4: Status 404 returned error can't find the container with id ddc345259f07efb7343fb36cceb4782fc946256559fd8f48cbc0d31bb9c914c4
Jul 30 21:52:25.659250 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.659161    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0d3803a0240ab6d48420082c32975e309f10be1bb85520caf46a66aa3edcdc"
Jul 30 21:52:25.659250 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.659184    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:25.660372 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.660344    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299451024" event={"ID":"10cdc8f9-34a7-4cb6-8522-c9120db1becf","Type":"ContainerStarted","Data":"ddc345259f07efb7343fb36cceb4782fc946256559fd8f48cbc0d31bb9c914c4"}
Jul 30 21:52:25.669723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:25.669686    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7240-139825299451024" oldPodUID="8e4983f7-c593-4547-bef6-afe403d8a66d" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf"
Jul 30 21:52:26.667447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:26.667412    2583 generic.go:356] "Generic (PLEG): container finished" podID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" containerID="b7503644b277781b01836e9a3c55b7a81de2025e662d748d8db3def228f85b38" exitCode=0
Jul 30 21:52:26.667838 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:26.667491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299451024" event={"ID":"10cdc8f9-34a7-4cb6-8522-c9120db1becf","Type":"ContainerDied","Data":"b7503644b277781b01836e9a3c55b7a81de2025e662d748d8db3def228f85b38"}
Jul 30 21:52:26.777061 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:26.777023    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e4983f7-c593-4547-bef6-afe403d8a66d" path="/var/lib/kubelet/pods/8e4983f7-c593-4547-bef6-afe403d8a66d/volumes"
Jul 30 21:52:27.815503 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.815478    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:27.833059 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.833032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299451024_10cdc8f9-34a7-4cb6-8522-c9120db1becf/test-curl-7240-139825299451024/0.log"
Jul 30 21:52:27.851720 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.851685    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:27.857476 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.857450    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299451024"]
Jul 30 21:52:27.911743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.911718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t\") pod \"10cdc8f9-34a7-4cb6-8522-c9120db1becf\" (UID: \"10cdc8f9-34a7-4cb6-8522-c9120db1becf\") "
Jul 30 21:52:27.913927 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.913894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t" (OuterVolumeSpecName: "kube-api-access-f4q4t") pod "10cdc8f9-34a7-4cb6-8522-c9120db1becf" (UID: "10cdc8f9-34a7-4cb6-8522-c9120db1becf"). InnerVolumeSpecName "kube-api-access-f4q4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:27.973746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.973665    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:27.974067 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.974052    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:27.974130 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.974072    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:27.974177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.974157    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" containerName="test-curl-7240-139825299451024"
Jul 30 21:52:27.977199 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.977177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:27.981028 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:27.981004    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:28.013391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.013364    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4q4t\" (UniqueName: \"kubernetes.io/projected/10cdc8f9-34a7-4cb6-8522-c9120db1becf-kube-api-access-f4q4t\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:28.114285 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.114252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhqr\" (UniqueName: \"kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr\") pod \"test-curl-7240-139825299349552\" (UID: \"0f941e33-7dda-4f75-ac6a-fd9678263653\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:28.215259 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.215223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhqr\" (UniqueName: \"kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr\") pod \"test-curl-7240-139825299349552\" (UID: \"0f941e33-7dda-4f75-ac6a-fd9678263653\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:28.223125 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.223094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhqr\" (UniqueName: \"kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr\") pod \"test-curl-7240-139825299349552\" (UID: \"0f941e33-7dda-4f75-ac6a-fd9678263653\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:28.289087 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.289053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:28.425954 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.425923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:28.429386 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:28.429359    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f941e33_7dda_4f75_ac6a_fd9678263653.slice/crio-3471c7ed00dc229885b21fb41c147cd24c7e8ba04c1999a139fb229c68c6d103 WatchSource:0}: Error finding container 3471c7ed00dc229885b21fb41c147cd24c7e8ba04c1999a139fb229c68c6d103: Status 404 returned error can't find the container with id 3471c7ed00dc229885b21fb41c147cd24c7e8ba04c1999a139fb229c68c6d103
Jul 30 21:52:28.679867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.679772    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:28.679867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.679783    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc345259f07efb7343fb36cceb4782fc946256559fd8f48cbc0d31bb9c914c4"
Jul 30 21:52:28.681063 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.681035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299349552" event={"ID":"0f941e33-7dda-4f75-ac6a-fd9678263653","Type":"ContainerStarted","Data":"3471c7ed00dc229885b21fb41c147cd24c7e8ba04c1999a139fb229c68c6d103"}
Jul 30 21:52:28.690079 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:28.690050    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299451024\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:28.774696 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:28.774659    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299451024\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" pod="opendatahub/test-curl-7240-139825299451024"
Jul 30 21:52:28.778446 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:28.778419    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10cdc8f9-34a7-4cb6-8522-c9120db1becf" path="/var/lib/kubelet/pods/10cdc8f9-34a7-4cb6-8522-c9120db1becf/volumes"
Jul 30 21:52:29.688137 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:29.688100    2583 generic.go:356] "Generic (PLEG): container finished" podID="0f941e33-7dda-4f75-ac6a-fd9678263653" containerID="94c75a4821d7d9f457ab703513827b67c4e73aa3021bb30f975807d659c1a20a" exitCode=0
Jul 30 21:52:29.688593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:29.688185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299349552" event={"ID":"0f941e33-7dda-4f75-ac6a-fd9678263653","Type":"ContainerDied","Data":"94c75a4821d7d9f457ab703513827b67c4e73aa3021bb30f975807d659c1a20a"}
Jul 30 21:52:30.263929 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.263887    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" podUID="42b29916-e90a-42ff-a945-87a2b315d5c2" containerName="main" containerID="cri-o://b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9" gracePeriod=45
Jul 30 21:52:30.525545 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.525489    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:52:30.636532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dknp\" (UniqueName: \"kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636670    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache\") pod \"42b29916-e90a-42ff-a945-87a2b315d5c2\" (UID: \"42b29916-e90a-42ff-a945-87a2b315d5c2\") "
Jul 30 21:52:30.636982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636828    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:52:30.636982 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home" (OuterVolumeSpecName: "home") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:52:30.637091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.636984    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-tmp-dir\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.637091 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.637045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:52:30.638990 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.638963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:52:30.638990 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.638983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp" (OuterVolumeSpecName: "kube-api-access-6dknp") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "kube-api-access-6dknp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:30.639128 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.638986    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm" (OuterVolumeSpecName: "dshm") pod "42b29916-e90a-42ff-a945-87a2b315d5c2" (UID: "42b29916-e90a-42ff-a945-87a2b315d5c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 30 21:52:30.694446 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.694416    2583 generic.go:356] "Generic (PLEG): container finished" podID="42b29916-e90a-42ff-a945-87a2b315d5c2" containerID="b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9" exitCode=0
Jul 30 21:52:30.694881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.694465    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"
Jul 30 21:52:30.694881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.694501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" event={"ID":"42b29916-e90a-42ff-a945-87a2b315d5c2","Type":"ContainerDied","Data":"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"}
Jul 30 21:52:30.694881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.694543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w" event={"ID":"42b29916-e90a-42ff-a945-87a2b315d5c2","Type":"ContainerDied","Data":"034f761c42ccc097913a3d1ec6d2435e57f4d41f71c41d5584de46e642bca5ec"}
Jul 30 21:52:30.694881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.694561    2583 scope.go:117] "RemoveContainer" containerID="b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"
Jul 30 21:52:30.704717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.704697    2583 scope.go:117] "RemoveContainer" containerID="b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"
Jul 30 21:52:30.705007 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:30.704984    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9\": container with ID starting with b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9 not found: ID does not exist" containerID="b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"
Jul 30 21:52:30.705121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.705020    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9"} err="failed to get container status \"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9\": rpc error: code = NotFound desc = could not find container \"b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9\": container with ID starting with b9b3665582d2ea70779b2955a2e0b734010c10dcd8f7b2fb5dddb7f17304b2a9 not found: ID does not exist"
Jul 30 21:52:30.715074 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.715044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:52:30.721351 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.721320    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-592c1e/rate-test-model-592c1e-kserve-84c9c6f5f5-sbv8w"]
Jul 30 21:52:30.737619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.737598    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-home\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.737619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.737619    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-dshm\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.737768 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.737628    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42b29916-e90a-42ff-a945-87a2b315d5c2-tls-certs\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.737768 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.737636    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42b29916-e90a-42ff-a945-87a2b315d5c2-model-cache\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.737768 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.737644    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dknp\" (UniqueName: \"kubernetes.io/projected/42b29916-e90a-42ff-a945-87a2b315d5c2-kube-api-access-6dknp\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:30.779282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.779204    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b29916-e90a-42ff-a945-87a2b315d5c2" path="/var/lib/kubelet/pods/42b29916-e90a-42ff-a945-87a2b315d5c2/volumes"
Jul 30 21:52:30.826689 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.826666    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:30.845295 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.845270    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299349552_0f941e33-7dda-4f75-ac6a-fd9678263653/test-curl-7240-139825299349552/0.log"
Jul 30 21:52:30.865353 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.865323    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:30.868921 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.868897    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:30.939424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.939390    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dhqr\" (UniqueName: \"kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr\") pod \"0f941e33-7dda-4f75-ac6a-fd9678263653\" (UID: \"0f941e33-7dda-4f75-ac6a-fd9678263653\") "
Jul 30 21:52:30.941674 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.941649    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr" (OuterVolumeSpecName: "kube-api-access-4dhqr") pod "0f941e33-7dda-4f75-ac6a-fd9678263653" (UID: "0f941e33-7dda-4f75-ac6a-fd9678263653"). InnerVolumeSpecName "kube-api-access-4dhqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:30.985002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.984975    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:30.985364 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985350    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f941e33-7dda-4f75-ac6a-fd9678263653" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:30.985424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985369    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f941e33-7dda-4f75-ac6a-fd9678263653" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:30.985424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985387    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b29916-e90a-42ff-a945-87a2b315d5c2" containerName="main"
Jul 30 21:52:30.985424 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985392    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b29916-e90a-42ff-a945-87a2b315d5c2" containerName="main"
Jul 30 21:52:30.985513 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985463    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f941e33-7dda-4f75-ac6a-fd9678263653" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:30.985513 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.985475    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b29916-e90a-42ff-a945-87a2b315d5c2" containerName="main"
Jul 30 21:52:30.988494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.988479    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:30.994040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:30.994017    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:31.040850 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.040777    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhqr\" (UniqueName: \"kubernetes.io/projected/0f941e33-7dda-4f75-ac6a-fd9678263653-kube-api-access-4dhqr\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:31.142196 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.142154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9g7h\" (UniqueName: \"kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h\") pod \"test-curl-7240-139825299349552\" (UID: \"dd140854-fb16-4ada-9a02-d03f82f8558a\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:31.242911 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.242878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9g7h\" (UniqueName: \"kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h\") pod \"test-curl-7240-139825299349552\" (UID: \"dd140854-fb16-4ada-9a02-d03f82f8558a\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:31.250249 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.250223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9g7h\" (UniqueName: \"kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h\") pod \"test-curl-7240-139825299349552\" (UID: \"dd140854-fb16-4ada-9a02-d03f82f8558a\") " pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:31.298694 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.298622    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:31.431963 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.431934    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:31.434325 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:31.434296    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd140854_fb16_4ada_9a02_d03f82f8558a.slice/crio-5f8edbc65be3768cea4e6bd833d714afe14e383c6c02fab8f67e7d734357d495 WatchSource:0}: Error finding container 5f8edbc65be3768cea4e6bd833d714afe14e383c6c02fab8f67e7d734357d495: Status 404 returned error can't find the container with id 5f8edbc65be3768cea4e6bd833d714afe14e383c6c02fab8f67e7d734357d495
Jul 30 21:52:31.701377 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.701295    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:31.701377 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.701305    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3471c7ed00dc229885b21fb41c147cd24c7e8ba04c1999a139fb229c68c6d103"
Jul 30 21:52:31.702624 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.702599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299349552" event={"ID":"dd140854-fb16-4ada-9a02-d03f82f8558a","Type":"ContainerStarted","Data":"5f8edbc65be3768cea4e6bd833d714afe14e383c6c02fab8f67e7d734357d495"}
Jul 30 21:52:31.712619 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:31.712593    2583 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7240-139825299349552" oldPodUID="0f941e33-7dda-4f75-ac6a-fd9678263653" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a"
Jul 30 21:52:32.709628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:32.709516    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd140854-fb16-4ada-9a02-d03f82f8558a" containerID="54d37298652c64471479019352618d05a92925377776649ba3e9a21f3c273393" exitCode=0
Jul 30 21:52:32.709628 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:32.709608    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-139825299349552" event={"ID":"dd140854-fb16-4ada-9a02-d03f82f8558a","Type":"ContainerDied","Data":"54d37298652c64471479019352618d05a92925377776649ba3e9a21f3c273393"}
Jul 30 21:52:32.777060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:32.777026    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f941e33-7dda-4f75-ac6a-fd9678263653" path="/var/lib/kubelet/pods/0f941e33-7dda-4f75-ac6a-fd9678263653/volumes"
Jul 30 21:52:33.860370 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.860345    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:33.885270 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.885236    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7240-139825299349552_dd140854-fb16-4ada-9a02-d03f82f8558a/test-curl-7240-139825299349552/0.log"
Jul 30 21:52:33.906180 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.906144    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:33.911866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.911840    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-139825299349552"]
Jul 30 21:52:33.965062 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.964968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9g7h\" (UniqueName: \"kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h\") pod \"dd140854-fb16-4ada-9a02-d03f82f8558a\" (UID: \"dd140854-fb16-4ada-9a02-d03f82f8558a\") "
Jul 30 21:52:33.967229 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:33.967188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h" (OuterVolumeSpecName: "kube-api-access-n9g7h") pod "dd140854-fb16-4ada-9a02-d03f82f8558a" (UID: "dd140854-fb16-4ada-9a02-d03f82f8558a"). InnerVolumeSpecName "kube-api-access-n9g7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:52:34.065755 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:34.065714    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9g7h\" (UniqueName: \"kubernetes.io/projected/dd140854-fb16-4ada-9a02-d03f82f8558a-kube-api-access-n9g7h\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:52:34.722098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:34.722064    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:34.722098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:34.722094    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f8edbc65be3768cea4e6bd833d714afe14e383c6c02fab8f67e7d734357d495"
Jul 30 21:52:34.734631 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:52:34.734600    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7240-139825299349552\" is forbidden: User \"system:node:ip-10-0-143-139.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-139.ec2.internal' and this object" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a" pod="opendatahub/test-curl-7240-139825299349552"
Jul 30 21:52:34.780379 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:34.780346    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a" path="/var/lib/kubelet/pods/dd140854-fb16-4ada-9a02-d03f82f8558a/volumes"
Jul 30 21:52:35.450494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:35.450457    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:35.501731 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:35.501693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:37.632503 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:37.632460    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:37.726681 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:37.726651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:38.720476 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.720442    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:52:38.720939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.720922    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:38.720988 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.720942    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:38.721029 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.721007    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd140854-fb16-4ada-9a02-d03f82f8558a" containerName="test-curl-7240-139825299349552"
Jul 30 21:52:38.724257 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.724241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.726304 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.726283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-df94bf-serving-cert\""
Jul 30 21:52:38.732448 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.732424    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:52:38.789767 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.789716    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:38.810700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.810665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.810873 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.810732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn4k\" (UniqueName: \"kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.810873 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.810803    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.911649 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.911613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.911867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.911665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.911867 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.911702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn4k\" (UniqueName: \"kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.912362 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.912339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.914273 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.914248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.920207 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.920183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn4k\" (UniqueName: \"kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k\") pod \"maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:38.941829 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:38.941798    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:39.036330 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.036295    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:39.181505 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.181471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:52:39.184859 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:39.184822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2660b083_ca23_4f42_ace9_75d059e1e5c9.slice/crio-87900da2e3137caf3a649069978c9ec1f4acbdef4b7c55b67ac18481ec71e725 WatchSource:0}: Error finding container 87900da2e3137caf3a649069978c9ec1f4acbdef4b7c55b67ac18481ec71e725: Status 404 returned error can't find the container with id 87900da2e3137caf3a649069978c9ec1f4acbdef4b7c55b67ac18481ec71e725
Jul 30 21:52:39.357175 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.357090    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:39.449068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.449034    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:52:39.451889 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.451859    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:52:39.456677 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.456654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.458620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.458593    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-df94bf\""
Jul 30 21:52:39.466532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.466492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:52:39.489745 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.489699    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:52:39.497938 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.497899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.502635 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.501874    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-df94bf-dockercfg-4sxrq\""
Jul 30 21:52:39.507148 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.507118    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:52:39.617194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.617097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.617194 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.617164    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vk4z\" (UniqueName: \"kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.617391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.617261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.617391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.617300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxpnf\" (UniqueName: \"kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.718589 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.718515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.718589 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.718591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vk4z\" (UniqueName: \"kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.718852 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.718637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.718852 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.718657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxpnf\" (UniqueName: \"kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.719429 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.719394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.719561 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.719528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.726443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.726419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxpnf\" (UniqueName: \"kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf\") pod \"payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.727075 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.727058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vk4z\" (UniqueName: \"kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z\") pod \"payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.750085 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.750055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" event={"ID":"2660b083-ca23-4f42-ace9-75d059e1e5c9","Type":"ContainerStarted","Data":"87900da2e3137caf3a649069978c9ec1f4acbdef4b7c55b67ac18481ec71e725"}
Jul 30 21:52:39.768964 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.768939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:39.813211 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.813175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:39.926265 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:39.925209    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:52:39.926701 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:39.926674    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2535339b_1cb5_4dfd_9797_82b9fc0c486f.slice/crio-5ce0f78c60343e1bc00572211168920d497e5a71b3dbf04badece7198df21b89 WatchSource:0}: Error finding container 5ce0f78c60343e1bc00572211168920d497e5a71b3dbf04badece7198df21b89: Status 404 returned error can't find the container with id 5ce0f78c60343e1bc00572211168920d497e5a71b3dbf04badece7198df21b89
Jul 30 21:52:40.180101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.180080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:52:40.182184 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:52:40.182145    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd60b83a_7462_46fb_8ab4_274029b33914.slice/crio-84e321b39fa101d36eb61d02c99b3cffca01c64ad2921e66a0ab8239bfb507b1 WatchSource:0}: Error finding container 84e321b39fa101d36eb61d02c99b3cffca01c64ad2921e66a0ab8239bfb507b1: Status 404 returned error can't find the container with id 84e321b39fa101d36eb61d02c99b3cffca01c64ad2921e66a0ab8239bfb507b1
Jul 30 21:52:40.757844 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.757809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" event={"ID":"2535339b-1cb5-4dfd-9797-82b9fc0c486f","Type":"ContainerStarted","Data":"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"}
Jul 30 21:52:40.758314 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.757865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" event={"ID":"2535339b-1cb5-4dfd-9797-82b9fc0c486f","Type":"ContainerStarted","Data":"5ce0f78c60343e1bc00572211168920d497e5a71b3dbf04badece7198df21b89"}
Jul 30 21:52:40.758314 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.757918    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:40.759371 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.759346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" event={"ID":"dd60b83a-7462-46fb-8ab4-274029b33914","Type":"ContainerStarted","Data":"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"}
Jul 30 21:52:40.759494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.759386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" event={"ID":"dd60b83a-7462-46fb-8ab4-274029b33914","Type":"ContainerStarted","Data":"84e321b39fa101d36eb61d02c99b3cffca01c64ad2921e66a0ab8239bfb507b1"}
Jul 30 21:52:40.759494 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.759460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:52:40.760975 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.760946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" event={"ID":"2660b083-ca23-4f42-ace9-75d059e1e5c9","Type":"ContainerStarted","Data":"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"}
Jul 30 21:52:40.761072 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.761029    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:40.772321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.772283    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" podStartSLOduration=1.772261196 podStartE2EDuration="1.772261196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:52:40.771309586 +0000 UTC m=+3332.537271465" watchObservedRunningTime="2026-07-30 21:52:40.772261196 +0000 UTC m=+3332.538223116"
Jul 30 21:52:40.788206 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.788153    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" podStartSLOduration=1.788139511 podStartE2EDuration="1.788139511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:52:40.784688907 +0000 UTC m=+3332.550650786" watchObservedRunningTime="2026-07-30 21:52:40.788139511 +0000 UTC m=+3332.554101394"
Jul 30 21:52:40.798626 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:40.798541    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" podStartSLOduration=2.059744814 podStartE2EDuration="2.798523356s" totalImagesPullingTime="738.778542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:52:40.797676557 +0000 UTC m=+3332.563638437" watchObservedRunningTime="2026-07-30 21:52:40.798523356 +0000 UTC m=+3332.564485236"
Jul 30 21:52:46.777399 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:46.777366    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:52:51.770237 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:51.770205    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:52:51.770721 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:52:51.770364    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:53:02.594991 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:02.594951    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:03.267850 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:03.267805    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:03.664084 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:03.663982    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:03.754610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:03.754531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:06.982556 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:06.982520    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:07.083269 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:07.083227    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:08.135544 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.135509    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:53:08.140754 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.140728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.142696 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.142676    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b274d1-serving-cert\""
Jul 30 21:53:08.147741 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.147720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:53:08.171197 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.171161    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:08.244161 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.244124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.244337 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.244179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfgv\" (UniqueName: \"kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.244337 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.244248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.345478 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.345443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.345683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.345495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfgv\" (UniqueName: \"kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.345683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.345527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.345683 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:53:08.345656    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-serving-cert: secret "maas-api-e2e-ipp-b-b274d1-serving-cert" not found
Jul 30 21:53:08.345791 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:53:08.345724    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls podName:f83c536a-748b-4976-a565-4cd67e8f78e1 nodeName:}" failed. No retries permitted until 2026-07-30 21:53:08.845707152 +0000 UTC m=+3360.611669008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls") pod "maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" (UID: "f83c536a-748b-4976-a565-4cd67e8f78e1") : secret "maas-api-e2e-ipp-b-b274d1-serving-cert" not found
Jul 30 21:53:08.346210 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.346188    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.353464 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.353434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfgv\" (UniqueName: \"kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.675147 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.675111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:08.775351 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.775266    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:08.789198 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.789157    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:53:08.792912 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.792892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.794957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.794937    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b274d1\""
Jul 30 21:53:08.795072 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.794940    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b274d1-dockercfg-xrjv4\""
Jul 30 21:53:08.800866 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.800842    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:53:08.822458 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.822423    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:53:08.826608 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.826589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.835094 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.835036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:53:08.850479 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.850451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:08.850660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.850500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.850660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.850526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zzx\" (UniqueName: \"kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.850660 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:53:08.850634    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-serving-cert: secret "maas-api-e2e-ipp-b-b274d1-serving-cert" not found
Jul 30 21:53:08.850660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.850651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.850875 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:53:08.850698    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls podName:f83c536a-748b-4976-a565-4cd67e8f78e1 nodeName:}" failed. No retries permitted until 2026-07-30 21:53:09.850678626 +0000 UTC m=+3361.616640527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls") pod "maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" (UID: "f83c536a-748b-4976-a565-4cd67e8f78e1") : secret "maas-api-e2e-ipp-b-b274d1-serving-cert" not found
Jul 30 21:53:08.850875 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.850743    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkt92\" (UniqueName: \"kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.917680 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.917631    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:08.951890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.951805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.951890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.951846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zzx\" (UniqueName: \"kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.952099 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.951915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.952099 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.951957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt92\" (UniqueName: \"kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.952560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.952541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.952919 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.952894    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:08.959469 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.959441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zzx\" (UniqueName: \"kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx\") pod \"payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:08.959615 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:08.959502    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt92\" (UniqueName: \"kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92\") pod \"payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:09.105006 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.104966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:09.137784 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.137740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:09.294765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.294739    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:53:09.297040 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:53:09.297014    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1cb4a7_e2c9_443e_b38c_d75196ef92be.slice/crio-27f071ef58c53eb941b43f2b681530581160979118472c98a9c048764dcdb77f WatchSource:0}: Error finding container 27f071ef58c53eb941b43f2b681530581160979118472c98a9c048764dcdb77f: Status 404 returned error can't find the container with id 27f071ef58c53eb941b43f2b681530581160979118472c98a9c048764dcdb77f
Jul 30 21:53:09.479373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.465099    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:53:09.479373 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:53:09.467347    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d7bce1_113b_4a43_83fb_1044254f1fbd.slice/crio-febe871e7c61e964f590fa3e44a9686bbadeb7639d0da1d41d2f883c071e82fc WatchSource:0}: Error finding container febe871e7c61e964f590fa3e44a9686bbadeb7639d0da1d41d2f883c071e82fc: Status 404 returned error can't find the container with id febe871e7c61e964f590fa3e44a9686bbadeb7639d0da1d41d2f883c071e82fc
Jul 30 21:53:09.858617 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.858579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:09.861177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.861152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:09.926883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.926841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" event={"ID":"70d7bce1-113b-4a43-83fb-1044254f1fbd","Type":"ContainerStarted","Data":"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"}
Jul 30 21:53:09.927076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.926915    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:09.927076 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.926942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" event={"ID":"70d7bce1-113b-4a43-83fb-1044254f1fbd","Type":"ContainerStarted","Data":"febe871e7c61e964f590fa3e44a9686bbadeb7639d0da1d41d2f883c071e82fc"}
Jul 30 21:53:09.928493 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.928467    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" event={"ID":"2f1cb4a7-e2c9-443e-b38c-d75196ef92be","Type":"ContainerStarted","Data":"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"}
Jul 30 21:53:09.928636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.928517    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" event={"ID":"2f1cb4a7-e2c9-443e-b38c-d75196ef92be","Type":"ContainerStarted","Data":"27f071ef58c53eb941b43f2b681530581160979118472c98a9c048764dcdb77f"}
Jul 30 21:53:09.928636 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.928546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:09.941093 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.941050    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" podStartSLOduration=1.941036808 podStartE2EDuration="1.941036808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:53:09.940435088 +0000 UTC m=+3361.706396965" watchObservedRunningTime="2026-07-30 21:53:09.941036808 +0000 UTC m=+3361.706998687"
Jul 30 21:53:09.955156 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.955109    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" podStartSLOduration=1.955093233 podStartE2EDuration="1.955093233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:53:09.952925426 +0000 UTC m=+3361.718887307" watchObservedRunningTime="2026-07-30 21:53:09.955093233 +0000 UTC m=+3361.721055112"
Jul 30 21:53:09.955530 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:09.955496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:10.104749 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:10.104717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:53:10.109469 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:53:10.109353    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83c536a_748b_4976_a565_4cd67e8f78e1.slice/crio-d5a2b918fee792dbb099e64f1634620ed979494e21b4748490f7c0522b40773b WatchSource:0}: Error finding container d5a2b918fee792dbb099e64f1634620ed979494e21b4748490f7c0522b40773b: Status 404 returned error can't find the container with id d5a2b918fee792dbb099e64f1634620ed979494e21b4748490f7c0522b40773b
Jul 30 21:53:10.935875 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:10.935829    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" event={"ID":"f83c536a-748b-4976-a565-4cd67e8f78e1","Type":"ContainerStarted","Data":"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"}
Jul 30 21:53:10.936310 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:10.935893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" event={"ID":"f83c536a-748b-4976-a565-4cd67e8f78e1","Type":"ContainerStarted","Data":"d5a2b918fee792dbb099e64f1634620ed979494e21b4748490f7c0522b40773b"}
Jul 30 21:53:10.951622 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:10.951549    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" podStartSLOduration=2.655559415 podStartE2EDuration="2.951532819s" totalImagesPullingTime="295.973404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:53:10.949694722 +0000 UTC m=+3362.715656600" watchObservedRunningTime="2026-07-30 21:53:10.951532819 +0000 UTC m=+3362.717494698"
Jul 30 21:53:11.941237 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:11.941203    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:17.617839 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:17.617802    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:17.952912 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:17.952831    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:53:20.938066 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:20.938033    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:53:20.938553 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:20.938108    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:53:37.082345 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:37.082306    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:53:43.872821 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:43.872784    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:53:43.986871 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:43.986837    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr_2f1cb4a7-e2c9-443e-b38c-d75196ef92be/payload-processing/0.log"
Jul 30 21:53:51.984774 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:53:51.984738    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:28.694094 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:28.694004    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf_dd60b83a-7462-46fb-8ab4-274029b33914/payload-processing/0.log"
Jul 30 21:54:28.812182 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:28.812139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:54:28.938472 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:28.938443    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr_2f1cb4a7-e2c9-443e-b38c-d75196ef92be/payload-processing/0.log"
Jul 30 21:54:29.219458 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:29.219411    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:29.517819 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:29.517782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:29.679952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:29.679914    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:32.324533 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:32.324487    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:32.419750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:32.419708    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:33.437587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.437545    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:54:33.441463 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.441440    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.443548 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.443522    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7d83c0-serving-cert\""
Jul 30 21:54:33.451949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.451927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:54:33.473154 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.473115    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:33.599711 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.599675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.599711 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.599713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpwm9\" (UniqueName: \"kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.599920 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.599795    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.700323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.700230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.700323 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.700267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpwm9\" (UniqueName: \"kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.700542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.700328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.700542 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:54:33.700399    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-serving-cert: secret "maas-api-e2e-ipp-cleanup-7d83c0-serving-cert" not found
Jul 30 21:54:33.700542 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:54:33.700476    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls podName:d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0 nodeName:}" failed. No retries permitted until 2026-07-30 21:54:34.200455297 +0000 UTC m=+3445.966417153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" (UID: "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0") : secret "maas-api-e2e-ipp-cleanup-7d83c0-serving-cert" not found
Jul 30 21:54:33.700962 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.700945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.708022 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.708001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpwm9\" (UniqueName: \"kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:33.941462 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:33.941428    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:34.040841 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.040806    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:34.051953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.051920    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:54:34.055641 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.055624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.057664 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.057644    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7d83c0\""
Jul 30 21:54:34.057740 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.057683    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7d83c0-dockercfg-mh8gj\""
Jul 30 21:54:34.065540 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.065518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:54:34.145060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.145025    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:54:34.205048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.205017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:34.205260 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.205071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.205260 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.205111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkmw\" (UniqueName: \"kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.207728 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.207704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:34.306605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.306486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.306605 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.306546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkmw\" (UniqueName: \"kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.307532 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.307506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.313971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.313949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkmw\" (UniqueName: \"kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw\") pod \"payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.353945 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.353919    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:34.368121 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.368094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:34.519329 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.519297    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:54:34.520876 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:54:34.520822    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fcab2d_05f7_4ae4_9eae_cfdc8c90d6a0.slice/crio-1a8d0b3b1b94bf5c7d616f7ebdd6bc767d711e4668eeb2834b7dfc8a5b38a948 WatchSource:0}: Error finding container 1a8d0b3b1b94bf5c7d616f7ebdd6bc767d711e4668eeb2834b7dfc8a5b38a948: Status 404 returned error can't find the container with id 1a8d0b3b1b94bf5c7d616f7ebdd6bc767d711e4668eeb2834b7dfc8a5b38a948
Jul 30 21:54:34.545190 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:34.545163    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:54:34.546992 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:54:34.546959    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51bb544_05fe_42c7_8107_b229a947410a.slice/crio-13da6e43a38ddf23ab02f5a94f14ba7a266c078640ab8af6dc44d16828f4808e WatchSource:0}: Error finding container 13da6e43a38ddf23ab02f5a94f14ba7a266c078640ab8af6dc44d16828f4808e: Status 404 returned error can't find the container with id 13da6e43a38ddf23ab02f5a94f14ba7a266c078640ab8af6dc44d16828f4808e
Jul 30 21:54:35.400207 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.400164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" event={"ID":"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0","Type":"ContainerStarted","Data":"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"}
Jul 30 21:54:35.400420 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.400232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" event={"ID":"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0","Type":"ContainerStarted","Data":"1a8d0b3b1b94bf5c7d616f7ebdd6bc767d711e4668eeb2834b7dfc8a5b38a948"}
Jul 30 21:54:35.400420 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.400358    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:35.401883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.401852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" event={"ID":"c51bb544-05fe-42c7-8107-b229a947410a","Type":"ContainerStarted","Data":"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"}
Jul 30 21:54:35.402021 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.401903    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" event={"ID":"c51bb544-05fe-42c7-8107-b229a947410a","Type":"ContainerStarted","Data":"13da6e43a38ddf23ab02f5a94f14ba7a266c078640ab8af6dc44d16828f4808e"}
Jul 30 21:54:35.402021 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.401950    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:35.419055 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.418995    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" podStartSLOduration=1.987176658 podStartE2EDuration="2.41897707s" totalImagesPullingTime="431.800412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:54:35.417878915 +0000 UTC m=+3447.183840796" watchObservedRunningTime="2026-07-30 21:54:35.41897707 +0000 UTC m=+3447.184938982"
Jul 30 21:54:35.432561 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:35.432502    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" podStartSLOduration=1.432483962 podStartE2EDuration="1.432483962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:54:35.431611386 +0000 UTC m=+3447.197573256" watchObservedRunningTime="2026-07-30 21:54:35.432483962 +0000 UTC m=+3447.198445841"
Jul 30 21:54:41.414546 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:41.414510    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:54:46.410060 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:46.410018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:54:57.346287 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:54:57.346250    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:03.999593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:03.999539    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:55:04.000594 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.000534    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" podUID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" containerName="maas-api" containerID="cri-o://7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd" gracePeriod=30
Jul 30 21:55:04.018924 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.018891    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:04.063318 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.063283    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:55:04.064037 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.063978    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" podUID="c51bb544-05fe-42c7-8107-b229a947410a" containerName="payload-pre-processing" containerID="cri-o://a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049" gracePeriod=30
Jul 30 21:55:04.116777 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.116748    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:04.209631 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.209596    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:04.261922 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.261784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:04.328338 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.328309    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:55:04.356402 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.356374    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:55:04.363492 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.363469    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpwm9\" (UniqueName: \"kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9\") pod \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") "
Jul 30 21:55:04.363615 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.363595    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") pod \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") "
Jul 30 21:55:04.363674 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.363645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca\") pod \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\" (UID: \"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0\") "
Jul 30 21:55:04.364071 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.364041    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" (UID: "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:55:04.365865 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.365842    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" (UID: "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:55:04.365957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.365860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9" (OuterVolumeSpecName: "kube-api-access-zpwm9") pod "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" (UID: "d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0"). InnerVolumeSpecName "kube-api-access-zpwm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:55:04.464865 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.464831    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume\") pod \"c51bb544-05fe-42c7-8107-b229a947410a\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") "
Jul 30 21:55:04.465068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.464896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nkmw\" (UniqueName: \"kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw\") pod \"c51bb544-05fe-42c7-8107-b229a947410a\" (UID: \"c51bb544-05fe-42c7-8107-b229a947410a\") "
Jul 30 21:55:04.465068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.465032    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:04.465068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.465044    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:04.465068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.465054    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpwm9\" (UniqueName: \"kubernetes.io/projected/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0-kube-api-access-zpwm9\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:04.465356 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.465157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c51bb544-05fe-42c7-8107-b229a947410a" (UID: "c51bb544-05fe-42c7-8107-b229a947410a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:55:04.467148 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.467121    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw" (OuterVolumeSpecName: "kube-api-access-5nkmw") pod "c51bb544-05fe-42c7-8107-b229a947410a" (UID: "c51bb544-05fe-42c7-8107-b229a947410a"). InnerVolumeSpecName "kube-api-access-5nkmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:55:04.561950 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.561853    2583 generic.go:356] "Generic (PLEG): container finished" podID="c51bb544-05fe-42c7-8107-b229a947410a" containerID="a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049" exitCode=0
Jul 30 21:55:04.561950 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.561909    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"
Jul 30 21:55:04.562177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.561992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" event={"ID":"c51bb544-05fe-42c7-8107-b229a947410a","Type":"ContainerDied","Data":"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"}
Jul 30 21:55:04.562177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.562082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc" event={"ID":"c51bb544-05fe-42c7-8107-b229a947410a","Type":"ContainerDied","Data":"13da6e43a38ddf23ab02f5a94f14ba7a266c078640ab8af6dc44d16828f4808e"}
Jul 30 21:55:04.562177 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.562145    2583 scope.go:117] "RemoveContainer" containerID="a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"
Jul 30 21:55:04.562339 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.562139    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:04.563740 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.563702    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" containerID="7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd" exitCode=0
Jul 30 21:55:04.563857 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.563771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" event={"ID":"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0","Type":"ContainerDied","Data":"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"}
Jul 30 21:55:04.563857 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.563804    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s" event={"ID":"d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0","Type":"ContainerDied","Data":"1a8d0b3b1b94bf5c7d616f7ebdd6bc767d711e4668eeb2834b7dfc8a5b38a948"}
Jul 30 21:55:04.563936 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.563914    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"
Jul 30 21:55:04.565512 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.565491    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51bb544-05fe-42c7-8107-b229a947410a-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:04.565618 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.565518    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkmw\" (UniqueName: \"kubernetes.io/projected/c51bb544-05fe-42c7-8107-b229a947410a-kube-api-access-5nkmw\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:04.573367 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.573338    2583 scope.go:117] "RemoveContainer" containerID="a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"
Jul 30 21:55:04.573638 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:55:04.573616    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049\": container with ID starting with a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049 not found: ID does not exist" containerID="a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"
Jul 30 21:55:04.573729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.573648    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049"} err="failed to get container status \"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049\": rpc error: code = NotFound desc = could not find container \"a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049\": container with ID starting with a974535606dd4c3650e9dc25d8dbb8d8aa69c7d87edd23b11bbd9a1849057049 not found: ID does not exist"
Jul 30 21:55:04.573729 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.573671    2583 scope.go:117] "RemoveContainer" containerID="7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"
Jul 30 21:55:04.591600 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.583401    2583 scope.go:117] "RemoveContainer" containerID="7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"
Jul 30 21:55:04.591600 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:55:04.589697    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd\": container with ID starting with 7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd not found: ID does not exist" containerID="7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"
Jul 30 21:55:04.591600 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.589734    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd"} err="failed to get container status \"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd\": rpc error: code = NotFound desc = could not find container \"7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd\": container with ID starting with 7cd14e9570e1c993461e39ae5cd3d0bebc27b8698b5baab065a3bcebc731cdcd not found: ID does not exist"
Jul 30 21:55:04.592762 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.592739    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:55:04.594665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.594646    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7d83c0-7df6764d7b-547hc"]
Jul 30 21:55:04.597062 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.597043    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:55:04.607979 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.607959    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7d83c0-74c9dc7ff7-7sh6s"]
Jul 30 21:55:04.777352 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.777307    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c51bb544-05fe-42c7-8107-b229a947410a" path="/var/lib/kubelet/pods/c51bb544-05fe-42c7-8107-b229a947410a/volumes"
Jul 30 21:55:04.777815 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:04.777791    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" path="/var/lib/kubelet/pods/d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0/volumes"
Jul 30 21:55:10.415476 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:10.415436    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:10.468326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:10.468281    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:10.659151 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:10.659117    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:10.897150 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:10.897108    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:29.814534 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.814437    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:55:29.815090 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.814795    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" podUID="2660b083-ca23-4f42-ace9-75d059e1e5c9" containerName="maas-api" containerID="cri-o://393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a" gracePeriod=30
Jul 30 21:55:29.834803 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.834771    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:29.868766 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.868337    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:55:29.868939 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.868870    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" podUID="dd60b83a-7462-46fb-8ab4-274029b33914" containerName="payload-processing" containerID="cri-o://e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2" gracePeriod=30
Jul 30 21:55:29.875806 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.875776    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:55:29.876247 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.876070    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" podUID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" containerName="payload-pre-processing" containerID="cri-o://00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2" gracePeriod=30
Jul 30 21:55:29.910008 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.909979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:29.968624 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:29.968583    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:30.030860 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.030729    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:30.171794 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.171754    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:55:30.203666 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.203640    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:55:30.211256 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.211222    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:55:30.289875 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.289842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls\") pod \"2660b083-ca23-4f42-ace9-75d059e1e5c9\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.289893    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume\") pod \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.289926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume\") pod \"dd60b83a-7462-46fb-8ab4-274029b33914\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.289953    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca\") pod \"2660b083-ca23-4f42-ace9-75d059e1e5c9\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.289994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vk4z\" (UniqueName: \"kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z\") pod \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\" (UID: \"2535339b-1cb5-4dfd-9797-82b9fc0c486f\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxpnf\" (UniqueName: \"kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf\") pod \"dd60b83a-7462-46fb-8ab4-274029b33914\" (UID: \"dd60b83a-7462-46fb-8ab4-274029b33914\") "
Jul 30 21:55:30.290064 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkn4k\" (UniqueName: \"kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k\") pod \"2660b083-ca23-4f42-ace9-75d059e1e5c9\" (UID: \"2660b083-ca23-4f42-ace9-75d059e1e5c9\") "
Jul 30 21:55:30.290391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2535339b-1cb5-4dfd-9797-82b9fc0c486f" (UID: "2535339b-1cb5-4dfd-9797-82b9fc0c486f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:55:30.290391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2660b083-ca23-4f42-ace9-75d059e1e5c9" (UID: "2660b083-ca23-4f42-ace9-75d059e1e5c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:55:30.290391 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290352    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dd60b83a-7462-46fb-8ab4-274029b33914" (UID: "dd60b83a-7462-46fb-8ab4-274029b33914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:55:30.290645 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290532    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2535339b-1cb5-4dfd-9797-82b9fc0c486f-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.290645 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290593    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd60b83a-7462-46fb-8ab4-274029b33914-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.290645 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.290610    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2660b083-ca23-4f42-ace9-75d059e1e5c9-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.292316 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.292276    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2660b083-ca23-4f42-ace9-75d059e1e5c9" (UID: "2660b083-ca23-4f42-ace9-75d059e1e5c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:55:30.292449 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.292353    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf" (OuterVolumeSpecName: "kube-api-access-jxpnf") pod "dd60b83a-7462-46fb-8ab4-274029b33914" (UID: "dd60b83a-7462-46fb-8ab4-274029b33914"). InnerVolumeSpecName "kube-api-access-jxpnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:55:30.292449 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.292411    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k" (OuterVolumeSpecName: "kube-api-access-lkn4k") pod "2660b083-ca23-4f42-ace9-75d059e1e5c9" (UID: "2660b083-ca23-4f42-ace9-75d059e1e5c9"). InnerVolumeSpecName "kube-api-access-lkn4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:55:30.292800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.292782    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z" (OuterVolumeSpecName: "kube-api-access-8vk4z") pod "2535339b-1cb5-4dfd-9797-82b9fc0c486f" (UID: "2535339b-1cb5-4dfd-9797-82b9fc0c486f"). InnerVolumeSpecName "kube-api-access-8vk4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:55:30.391665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.391585    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vk4z\" (UniqueName: \"kubernetes.io/projected/2535339b-1cb5-4dfd-9797-82b9fc0c486f-kube-api-access-8vk4z\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.391665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.391624    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxpnf\" (UniqueName: \"kubernetes.io/projected/dd60b83a-7462-46fb-8ab4-274029b33914-kube-api-access-jxpnf\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.391665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.391639    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkn4k\" (UniqueName: \"kubernetes.io/projected/2660b083-ca23-4f42-ace9-75d059e1e5c9-kube-api-access-lkn4k\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.391665 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.391653    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2660b083-ca23-4f42-ace9-75d059e1e5c9-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:55:30.412994 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.412948    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:30.709746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.709652    2583 generic.go:356] "Generic (PLEG): container finished" podID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" containerID="00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2" exitCode=0
Jul 30 21:55:30.709746 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.709711    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"
Jul 30 21:55:30.709971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.709739    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" event={"ID":"2535339b-1cb5-4dfd-9797-82b9fc0c486f","Type":"ContainerDied","Data":"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"}
Jul 30 21:55:30.709971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.709800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l" event={"ID":"2535339b-1cb5-4dfd-9797-82b9fc0c486f","Type":"ContainerDied","Data":"5ce0f78c60343e1bc00572211168920d497e5a71b3dbf04badece7198df21b89"}
Jul 30 21:55:30.709971 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.709823    2583 scope.go:117] "RemoveContainer" containerID="00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"
Jul 30 21:55:30.711152 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.711123    2583 generic.go:356] "Generic (PLEG): container finished" podID="dd60b83a-7462-46fb-8ab4-274029b33914" containerID="e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2" exitCode=0
Jul 30 21:55:30.711293 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.711193    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"
Jul 30 21:55:30.711293 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.711205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" event={"ID":"dd60b83a-7462-46fb-8ab4-274029b33914","Type":"ContainerDied","Data":"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"}
Jul 30 21:55:30.711293 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.711251    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf" event={"ID":"dd60b83a-7462-46fb-8ab4-274029b33914","Type":"ContainerDied","Data":"84e321b39fa101d36eb61d02c99b3cffca01c64ad2921e66a0ab8239bfb507b1"}
Jul 30 21:55:30.712583 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.712535    2583 generic.go:356] "Generic (PLEG): container finished" podID="2660b083-ca23-4f42-ace9-75d059e1e5c9" containerID="393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a" exitCode=0
Jul 30 21:55:30.712704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.712596    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"
Jul 30 21:55:30.712704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.712628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" event={"ID":"2660b083-ca23-4f42-ace9-75d059e1e5c9","Type":"ContainerDied","Data":"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"}
Jul 30 21:55:30.712704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.712675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228" event={"ID":"2660b083-ca23-4f42-ace9-75d059e1e5c9","Type":"ContainerDied","Data":"87900da2e3137caf3a649069978c9ec1f4acbdef4b7c55b67ac18481ec71e725"}
Jul 30 21:55:30.720068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.720044    2583 scope.go:117] "RemoveContainer" containerID="00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"
Jul 30 21:55:30.720393 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:55:30.720366    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2\": container with ID starting with 00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2 not found: ID does not exist" containerID="00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"
Jul 30 21:55:30.720483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.720410    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2"} err="failed to get container status \"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2\": rpc error: code = NotFound desc = could not find container \"00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2\": container with ID starting with 00855f7e8d9650047dc1f7e4263c7622ceabc60239aa7914673d4d673a3f2cd2 not found: ID does not exist"
Jul 30 21:55:30.720483 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.720434    2583 scope.go:117] "RemoveContainer" containerID="e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"
Jul 30 21:55:30.731114 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.731100    2583 scope.go:117] "RemoveContainer" containerID="e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"
Jul 30 21:55:30.731391 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:55:30.731373    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2\": container with ID starting with e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2 not found: ID does not exist" containerID="e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"
Jul 30 21:55:30.731443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.731405    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2"} err="failed to get container status \"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2\": rpc error: code = NotFound desc = could not find container \"e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2\": container with ID starting with e4e65bca568ed43c666f042982ae13a25e3b9e95e099f0fdc7c55ea73c7764b2 not found: ID does not exist"
Jul 30 21:55:30.731443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.731423    2583 scope.go:117] "RemoveContainer" containerID="393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"
Jul 30 21:55:30.737697 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.737676    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:55:30.740098 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.740074    2583 scope.go:117] "RemoveContainer" containerID="393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"
Jul 30 21:55:30.740370 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:55:30.740344    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a\": container with ID starting with 393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a not found: ID does not exist" containerID="393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"
Jul 30 21:55:30.740428 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.740376    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a"} err="failed to get container status \"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a\": rpc error: code = NotFound desc = could not find container \"393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a\": container with ID starting with 393e78f1b01a2ff61d89852403807d59e275da08557c667f49cd3185ae269b1a not found: ID does not exist"
Jul 30 21:55:30.742113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.742096    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-df94bf-7f5cc4757b-qxr7l"]
Jul 30 21:55:30.751315 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.751287    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:55:30.757740 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.757715    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df94bf-6685df5c5d-zz228"]
Jul 30 21:55:30.765373 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.765349    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:55:30.768916 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.768896    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df94bf-6fbd4549cb-kw6jf"]
Jul 30 21:55:30.777048 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.777021    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" path="/var/lib/kubelet/pods/2535339b-1cb5-4dfd-9797-82b9fc0c486f/volumes"
Jul 30 21:55:30.777466 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.777448    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2660b083-ca23-4f42-ace9-75d059e1e5c9" path="/var/lib/kubelet/pods/2660b083-ca23-4f42-ace9-75d059e1e5c9/volumes"
Jul 30 21:55:30.777847 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:30.777829    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd60b83a-7462-46fb-8ab4-274029b33914" path="/var/lib/kubelet/pods/dd60b83a-7462-46fb-8ab4-274029b33914/volumes"
Jul 30 21:55:35.495579 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:35.495539    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:50.708021 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:50.707983    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:50.763347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:50.763312    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:55:53.876856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:55:53.876813    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:12.040224 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:12.040182    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:12.422116 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:12.422026    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:17.840328 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:17.840281    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:17.896890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:17.896854    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:18.147780 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:18.147676    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:18.376610 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:18.376545    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:28.946326 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:28.946289    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:56:28.946846 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:28.946644    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" podUID="f83c536a-748b-4976-a565-4cd67e8f78e1" containerName="maas-api" containerID="cri-o://63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48" gracePeriod=30
Jul 30 21:56:28.971001 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:28.970974    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:29.000428 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.000389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:56:29.000700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.000668    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" podUID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" containerName="payload-processing" containerID="cri-o://e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db" gracePeriod=30
Jul 30 21:56:29.012362 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.012331    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:56:29.012845 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.012810    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" podUID="70d7bce1-113b-4a43-83fb-1044254f1fbd" containerName="payload-pre-processing" containerID="cri-o://80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777" gracePeriod=30
Jul 30 21:56:29.047212 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.047173    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:29.133979 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.133825    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:29.235534 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.235493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:29.274016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.273982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:56:29.310828 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.310801    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:56:29.317181 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.317159    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:56:29.365026 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.364991    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca\") pod \"f83c536a-748b-4976-a565-4cd67e8f78e1\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") "
Jul 30 21:56:29.365209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume\") pod \"70d7bce1-113b-4a43-83fb-1044254f1fbd\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") "
Jul 30 21:56:29.365209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qfgv\" (UniqueName: \"kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv\") pod \"f83c536a-748b-4976-a565-4cd67e8f78e1\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") "
Jul 30 21:56:29.365209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365114    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") pod \"f83c536a-748b-4976-a565-4cd67e8f78e1\" (UID: \"f83c536a-748b-4976-a565-4cd67e8f78e1\") "
Jul 30 21:56:29.365209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365138    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58zzx\" (UniqueName: \"kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx\") pod \"70d7bce1-113b-4a43-83fb-1044254f1fbd\" (UID: \"70d7bce1-113b-4a43-83fb-1044254f1fbd\") "
Jul 30 21:56:29.365209 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365193    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkt92\" (UniqueName: \"kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92\") pod \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") "
Jul 30 21:56:29.365468 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365251    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume\") pod \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\" (UID: \"2f1cb4a7-e2c9-443e-b38c-d75196ef92be\") "
Jul 30 21:56:29.365691 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365660    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70d7bce1-113b-4a43-83fb-1044254f1fbd" (UID: "70d7bce1-113b-4a43-83fb-1044254f1fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:29.365804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365754    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f83c536a-748b-4976-a565-4cd67e8f78e1" (UID: "f83c536a-748b-4976-a565-4cd67e8f78e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:29.365804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.365772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f1cb4a7-e2c9-443e-b38c-d75196ef92be" (UID: "2f1cb4a7-e2c9-443e-b38c-d75196ef92be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:29.367347 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.367315    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f83c536a-748b-4976-a565-4cd67e8f78e1" (UID: "f83c536a-748b-4976-a565-4cd67e8f78e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:56:29.367498 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.367472    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv" (OuterVolumeSpecName: "kube-api-access-9qfgv") pod "f83c536a-748b-4976-a565-4cd67e8f78e1" (UID: "f83c536a-748b-4976-a565-4cd67e8f78e1"). InnerVolumeSpecName "kube-api-access-9qfgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:29.367559 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.367512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx" (OuterVolumeSpecName: "kube-api-access-58zzx") pod "70d7bce1-113b-4a43-83fb-1044254f1fbd" (UID: "70d7bce1-113b-4a43-83fb-1044254f1fbd"). InnerVolumeSpecName "kube-api-access-58zzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:29.367559 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.367538    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92" (OuterVolumeSpecName: "kube-api-access-hkt92") pod "2f1cb4a7-e2c9-443e-b38c-d75196ef92be" (UID: "2f1cb4a7-e2c9-443e-b38c-d75196ef92be"). InnerVolumeSpecName "kube-api-access-hkt92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:29.466748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466712    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466742    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f83c536a-748b-4976-a565-4cd67e8f78e1-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466748 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466754    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70d7bce1-113b-4a43-83fb-1044254f1fbd-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466768    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfgv\" (UniqueName: \"kubernetes.io/projected/f83c536a-748b-4976-a565-4cd67e8f78e1-kube-api-access-9qfgv\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466784    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f83c536a-748b-4976-a565-4cd67e8f78e1-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466794    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58zzx\" (UniqueName: \"kubernetes.io/projected/70d7bce1-113b-4a43-83fb-1044254f1fbd-kube-api-access-58zzx\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.466984 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.466802    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkt92\" (UniqueName: \"kubernetes.io/projected/2f1cb4a7-e2c9-443e-b38c-d75196ef92be-kube-api-access-hkt92\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:29.556612 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:29.556558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:30.021830 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.021793    2583 generic.go:356] "Generic (PLEG): container finished" podID="f83c536a-748b-4976-a565-4cd67e8f78e1" containerID="63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48" exitCode=0
Jul 30 21:56:30.022241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.021845    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"
Jul 30 21:56:30.022241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.021884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" event={"ID":"f83c536a-748b-4976-a565-4cd67e8f78e1","Type":"ContainerDied","Data":"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"}
Jul 30 21:56:30.022241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.021940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2" event={"ID":"f83c536a-748b-4976-a565-4cd67e8f78e1","Type":"ContainerDied","Data":"d5a2b918fee792dbb099e64f1634620ed979494e21b4748490f7c0522b40773b"}
Jul 30 21:56:30.022241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.021963    2583 scope.go:117] "RemoveContainer" containerID="63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"
Jul 30 21:56:30.023465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.023208    2583 generic.go:356] "Generic (PLEG): container finished" podID="70d7bce1-113b-4a43-83fb-1044254f1fbd" containerID="80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777" exitCode=0
Jul 30 21:56:30.023465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.023269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"
Jul 30 21:56:30.023465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.023292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" event={"ID":"70d7bce1-113b-4a43-83fb-1044254f1fbd","Type":"ContainerDied","Data":"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"}
Jul 30 21:56:30.023465 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.023349    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c" event={"ID":"70d7bce1-113b-4a43-83fb-1044254f1fbd","Type":"ContainerDied","Data":"febe871e7c61e964f590fa3e44a9686bbadeb7639d0da1d41d2f883c071e82fc"}
Jul 30 21:56:30.025111 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.025086    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" containerID="e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db" exitCode=0
Jul 30 21:56:30.025235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.025122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" event={"ID":"2f1cb4a7-e2c9-443e-b38c-d75196ef92be","Type":"ContainerDied","Data":"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"}
Jul 30 21:56:30.025235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.025142    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"
Jul 30 21:56:30.025235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.025153    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr" event={"ID":"2f1cb4a7-e2c9-443e-b38c-d75196ef92be","Type":"ContainerDied","Data":"27f071ef58c53eb941b43f2b681530581160979118472c98a9c048764dcdb77f"}
Jul 30 21:56:30.032584 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.032548    2583 scope.go:117] "RemoveContainer" containerID="63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"
Jul 30 21:56:30.032915 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:30.032845    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48\": container with ID starting with 63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48 not found: ID does not exist" containerID="63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"
Jul 30 21:56:30.032915 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.032877    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48"} err="failed to get container status \"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48\": rpc error: code = NotFound desc = could not find container \"63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48\": container with ID starting with 63b971a7f3ea665a094c79936c36edd21f8d7341310ca710ac8fef5062cafa48 not found: ID does not exist"
Jul 30 21:56:30.032915 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.032901    2583 scope.go:117] "RemoveContainer" containerID="80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"
Jul 30 21:56:30.042262 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.042244    2583 scope.go:117] "RemoveContainer" containerID="80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"
Jul 30 21:56:30.042542 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:30.042519    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777\": container with ID starting with 80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777 not found: ID does not exist" containerID="80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"
Jul 30 21:56:30.042615 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.042552    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777"} err="failed to get container status \"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777\": rpc error: code = NotFound desc = could not find container \"80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777\": container with ID starting with 80fbe41ed9d48901ea9e250c8c3b84af6939e88ab0dd428838633e6ac9aa0777 not found: ID does not exist"
Jul 30 21:56:30.042615 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.042603    2583 scope.go:117] "RemoveContainer" containerID="e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"
Jul 30 21:56:30.049233 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.049209    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:56:30.051892 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.051875    2583 scope.go:117] "RemoveContainer" containerID="e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"
Jul 30 21:56:30.052156 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:30.052136    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db\": container with ID starting with e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db not found: ID does not exist" containerID="e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"
Jul 30 21:56:30.052221 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.052162    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db"} err="failed to get container status \"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db\": rpc error: code = NotFound desc = could not find container \"e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db\": container with ID starting with e3aea990d4b02b96dce977d6486f2a81fbebd1b04737377acd443340729647db not found: ID does not exist"
Jul 30 21:56:30.053166 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.053148    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b274d1-b9948b98b-xj8j2"]
Jul 30 21:56:30.061230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.061210    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:56:30.064799 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.064780    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b274d1-77f64c9d68-f7crr"]
Jul 30 21:56:30.073037 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.073015    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:56:30.079694 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.079667    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b274d1-7bcfb949b4-7nx7c"]
Jul 30 21:56:30.778375 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.778340    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" path="/var/lib/kubelet/pods/2f1cb4a7-e2c9-443e-b38c-d75196ef92be/volumes"
Jul 30 21:56:30.778705 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.778690    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d7bce1-113b-4a43-83fb-1044254f1fbd" path="/var/lib/kubelet/pods/70d7bce1-113b-4a43-83fb-1044254f1fbd/volumes"
Jul 30 21:56:30.779021 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:30.779009    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83c536a-748b-4976-a565-4cd67e8f78e1" path="/var/lib/kubelet/pods/f83c536a-748b-4976-a565-4cd67e8f78e1/volumes"
Jul 30 21:56:35.616542 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:35.616500    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:35.681706 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:35.681671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:35.885426 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:35.885327    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:36.022213 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022179    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"]
Jul 30 21:56:36.022634 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022618    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83c536a-748b-4976-a565-4cd67e8f78e1" containerName="maas-api"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022635    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83c536a-748b-4976-a565-4cd67e8f78e1" containerName="maas-api"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022646    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" containerName="maas-api"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022651    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" containerName="maas-api"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022662    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" containerName="payload-pre-processing"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022668    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" containerName="payload-pre-processing"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022680    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d7bce1-113b-4a43-83fb-1044254f1fbd" containerName="payload-pre-processing"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022686    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d7bce1-113b-4a43-83fb-1044254f1fbd" containerName="payload-pre-processing"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022695    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c51bb544-05fe-42c7-8107-b229a947410a" containerName="payload-pre-processing"
Jul 30 21:56:36.022700 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022700    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c51bb544-05fe-42c7-8107-b229a947410a" containerName="payload-pre-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022711    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2660b083-ca23-4f42-ace9-75d059e1e5c9" containerName="maas-api"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022716    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2660b083-ca23-4f42-ace9-75d059e1e5c9" containerName="maas-api"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022722    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd60b83a-7462-46fb-8ab4-274029b33914" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022727    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd60b83a-7462-46fb-8ab4-274029b33914" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022732    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022737    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022791    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2535339b-1cb5-4dfd-9797-82b9fc0c486f" containerName="payload-pre-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022799    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2660b083-ca23-4f42-ace9-75d059e1e5c9" containerName="maas-api"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022806    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd60b83a-7462-46fb-8ab4-274029b33914" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022811    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83c536a-748b-4976-a565-4cd67e8f78e1" containerName="maas-api"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022816    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c51bb544-05fe-42c7-8107-b229a947410a" containerName="payload-pre-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022822    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f1cb4a7-e2c9-443e-b38c-d75196ef92be" containerName="payload-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022827    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d7bce1-113b-4a43-83fb-1044254f1fbd" containerName="payload-pre-processing"
Jul 30 21:56:36.023002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.022832    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9fcab2d-05f7-4ae4-9eae-cfdc8c90d6a0" containerName="maas-api"
Jul 30 21:56:36.027061 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.027043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:36.028908 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.028885    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:56:36.031730 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.031707    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"]
Jul 30 21:56:36.127539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.127502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56\" (UID: \"6fbd8c30-44a5-4924-8d1f-9a41cad7d424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:36.228155 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.228122    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56\" (UID: \"6fbd8c30-44a5-4924-8d1f-9a41cad7d424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:36.228765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.228744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56\" (UID: \"6fbd8c30-44a5-4924-8d1f-9a41cad7d424\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:36.340176 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.340142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:36.485977 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:36.485893    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"]
Jul 30 21:56:36.489449 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:56:36.489403    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbd8c30_44a5_4924_8d1f_9a41cad7d424.slice/crio-64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455 WatchSource:0}: Error finding container 64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455: Status 404 returned error can't find the container with id 64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455
Jul 30 21:56:37.066556 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:37.066465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56" event={"ID":"6fbd8c30-44a5-4924-8d1f-9a41cad7d424","Type":"ContainerStarted","Data":"c07da194859d1b38b1b2b9e1cf2b562e05724893ec26d65be57e1b6b6375887a"}
Jul 30 21:56:37.066556 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:37.066516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56" event={"ID":"6fbd8c30-44a5-4924-8d1f-9a41cad7d424","Type":"ContainerStarted","Data":"64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455"}
Jul 30 21:56:37.083009 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:37.082960    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56" podStartSLOduration=1.082943521 podStartE2EDuration="1.082943521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:56:37.08087979 +0000 UTC m=+3568.846841670" watchObservedRunningTime="2026-07-30 21:56:37.082943521 +0000 UTC m=+3568.848905462"
Jul 30 21:56:38.073316 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:38.073273    2583 generic.go:356] "Generic (PLEG): container finished" podID="6fbd8c30-44a5-4924-8d1f-9a41cad7d424" containerID="c07da194859d1b38b1b2b9e1cf2b562e05724893ec26d65be57e1b6b6375887a" exitCode=0
Jul 30 21:56:38.073801 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:38.073380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56" event={"ID":"6fbd8c30-44a5-4924-8d1f-9a41cad7d424","Type":"ContainerDied","Data":"c07da194859d1b38b1b2b9e1cf2b562e05724893ec26d65be57e1b6b6375887a"}
Jul 30 21:56:39.216633 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:39.216603    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:39.355366 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:39.355271    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca\") pod \"6fbd8c30-44a5-4924-8d1f-9a41cad7d424\" (UID: \"6fbd8c30-44a5-4924-8d1f-9a41cad7d424\") "
Jul 30 21:56:39.355704 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:39.355681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fbd8c30-44a5-4924-8d1f-9a41cad7d424" (UID: "6fbd8c30-44a5-4924-8d1f-9a41cad7d424"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:39.455951 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:39.455911    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fbd8c30-44a5-4924-8d1f-9a41cad7d424-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:40.085659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:40.085618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56" event={"ID":"6fbd8c30-44a5-4924-8d1f-9a41cad7d424","Type":"ContainerDied","Data":"64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455"}
Jul 30 21:56:40.085659 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:40.085651    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"
Jul 30 21:56:40.085900 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:40.085677    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64174c7d548a317b901e89a27e74f482373f1ec126673f68c590bec75ee2b455"
Jul 30 21:56:43.300769 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:43.300735    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:43.402101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:43.402058    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:46.431202 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.431163    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:56:46.431736 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.431421    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" podUID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" containerName="maas-api" containerID="cri-o://2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6" gracePeriod=30
Jul 30 21:56:46.461607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.461558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:46.496942 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.496900    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:56:46.497157 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.497131    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" podUID="a779da23-21f4-4689-a35a-d32bcf97642a" containerName="payload-processing" containerID="cri-o://45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9" gracePeriod=30
Jul 30 21:56:46.503245 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.503201    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:56:46.503907 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.503433    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" podUID="4e718140-09a0-464d-bed5-c569885a92c2" containerName="payload-pre-processing" containerID="cri-o://d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f" gracePeriod=30
Jul 30 21:56:46.560148 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.560103    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:46.637489 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.637442    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"]
Jul 30 21:56:46.639520 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.639465    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:46.649792 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.646820    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2abe52-5wt56"]
Jul 30 21:56:46.797957 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.797554    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fbd8c30-44a5-4924-8d1f-9a41cad7d424" path="/var/lib/kubelet/pods/6fbd8c30-44a5-4924-8d1f-9a41cad7d424/volumes"
Jul 30 21:56:46.807789 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.807745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:46.814293 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.814250    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:56:46.817800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.817779    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:56:46.826313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.826293    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:56:46.915261 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2689k\" (UniqueName: \"kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k\") pod \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") "
Jul 30 21:56:46.915443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca\") pod \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") "
Jul 30 21:56:46.915443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc9sz\" (UniqueName: \"kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz\") pod \"4e718140-09a0-464d-bed5-c569885a92c2\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") "
Jul 30 21:56:46.915443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915380    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgl5h\" (UniqueName: \"kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h\") pod \"a779da23-21f4-4689-a35a-d32bcf97642a\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") "
Jul 30 21:56:46.915443 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume\") pod \"a779da23-21f4-4689-a35a-d32bcf97642a\" (UID: \"a779da23-21f4-4689-a35a-d32bcf97642a\") "
Jul 30 21:56:46.915702 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls\") pod \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\" (UID: \"c8c1718b-86d1-4f41-8ad3-dc8b1a092994\") "
Jul 30 21:56:46.915702 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume\") pod \"4e718140-09a0-464d-bed5-c569885a92c2\" (UID: \"4e718140-09a0-464d-bed5-c569885a92c2\") "
Jul 30 21:56:46.915848 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915814    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8c1718b-86d1-4f41-8ad3-dc8b1a092994" (UID: "c8c1718b-86d1-4f41-8ad3-dc8b1a092994"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:46.915980 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.915939    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a779da23-21f4-4689-a35a-d32bcf97642a" (UID: "a779da23-21f4-4689-a35a-d32bcf97642a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:46.916255 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.916092    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e718140-09a0-464d-bed5-c569885a92c2" (UID: "4e718140-09a0-464d-bed5-c569885a92c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:46.917717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.917689    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h" (OuterVolumeSpecName: "kube-api-access-mgl5h") pod "a779da23-21f4-4689-a35a-d32bcf97642a" (UID: "a779da23-21f4-4689-a35a-d32bcf97642a"). InnerVolumeSpecName "kube-api-access-mgl5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:46.917717 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.917698    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz" (OuterVolumeSpecName: "kube-api-access-rc9sz") pod "4e718140-09a0-464d-bed5-c569885a92c2" (UID: "4e718140-09a0-464d-bed5-c569885a92c2"). InnerVolumeSpecName "kube-api-access-rc9sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:46.918085 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.918068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k" (OuterVolumeSpecName: "kube-api-access-2689k") pod "c8c1718b-86d1-4f41-8ad3-dc8b1a092994" (UID: "c8c1718b-86d1-4f41-8ad3-dc8b1a092994"). InnerVolumeSpecName "kube-api-access-2689k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:56:46.918085 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:46.918072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8c1718b-86d1-4f41-8ad3-dc8b1a092994" (UID: "c8c1718b-86d1-4f41-8ad3-dc8b1a092994"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:56:47.016953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016916    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.016953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016946    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc9sz\" (UniqueName: \"kubernetes.io/projected/4e718140-09a0-464d-bed5-c569885a92c2-kube-api-access-rc9sz\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.016953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgl5h\" (UniqueName: \"kubernetes.io/projected/a779da23-21f4-4689-a35a-d32bcf97642a-kube-api-access-mgl5h\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.016953 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016964    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a779da23-21f4-4689-a35a-d32bcf97642a-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.017233 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016973    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.017233 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016982    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e718140-09a0-464d-bed5-c569885a92c2-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.017233 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.016991    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2689k\" (UniqueName: \"kubernetes.io/projected/c8c1718b-86d1-4f41-8ad3-dc8b1a092994-kube-api-access-2689k\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:47.037580 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.037528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:47.122349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.122320    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" containerID="2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6" exitCode=0
Jul 30 21:56:47.122539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.122364    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"
Jul 30 21:56:47.122539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.122412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" event={"ID":"c8c1718b-86d1-4f41-8ad3-dc8b1a092994","Type":"ContainerDied","Data":"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"}
Jul 30 21:56:47.122539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.122466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj" event={"ID":"c8c1718b-86d1-4f41-8ad3-dc8b1a092994","Type":"ContainerDied","Data":"f8ac929702a2f4a3374c9aea200204648b42671322a0399b607870d9d5038f65"}
Jul 30 21:56:47.122539 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.122495    2583 scope.go:117] "RemoveContainer" containerID="2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"
Jul 30 21:56:47.123883 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.123860    2583 generic.go:356] "Generic (PLEG): container finished" podID="a779da23-21f4-4689-a35a-d32bcf97642a" containerID="45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9" exitCode=0
Jul 30 21:56:47.124016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.123902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" event={"ID":"a779da23-21f4-4689-a35a-d32bcf97642a","Type":"ContainerDied","Data":"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"}
Jul 30 21:56:47.124016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.123945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q" event={"ID":"a779da23-21f4-4689-a35a-d32bcf97642a","Type":"ContainerDied","Data":"e2e7f3ae21ddf4fb6ecb81a7661c7ff697a303a01bcf9614d747d1ad47723719"}
Jul 30 21:56:47.124016 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.123905    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"
Jul 30 21:56:47.125747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.125719    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e718140-09a0-464d-bed5-c569885a92c2" containerID="d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f" exitCode=0
Jul 30 21:56:47.125865 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.125764    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"
Jul 30 21:56:47.125865 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.125810    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" event={"ID":"4e718140-09a0-464d-bed5-c569885a92c2","Type":"ContainerDied","Data":"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"}
Jul 30 21:56:47.125983 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.125873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh" event={"ID":"4e718140-09a0-464d-bed5-c569885a92c2","Type":"ContainerDied","Data":"e578033c6d4f74c467873212c452683c534ba1b1a3dd4994945caf013df25479"}
Jul 30 21:56:47.133952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.133719    2583 scope.go:117] "RemoveContainer" containerID="2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"
Jul 30 21:56:47.134085 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:47.134043    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6\": container with ID starting with 2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6 not found: ID does not exist" containerID="2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"
Jul 30 21:56:47.134133 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.134077    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6"} err="failed to get container status \"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6\": rpc error: code = NotFound desc = could not find container \"2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6\": container with ID starting with 2e0bff9d171d0a0d097c7a211f400bad71d4f00f50d2384ab9f7907bfe0b4dc6 not found: ID does not exist"
Jul 30 21:56:47.134133 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.134101    2583 scope.go:117] "RemoveContainer" containerID="45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"
Jul 30 21:56:47.143082 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.143064    2583 scope.go:117] "RemoveContainer" containerID="45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"
Jul 30 21:56:47.143345 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:47.143326    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9\": container with ID starting with 45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9 not found: ID does not exist" containerID="45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"
Jul 30 21:56:47.143401 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.143352    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9"} err="failed to get container status \"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9\": rpc error: code = NotFound desc = could not find container \"45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9\": container with ID starting with 45c8430a8062858165601a8e76cc3242df65f762ce8bd590ffd1736ac3bd4fd9 not found: ID does not exist"
Jul 30 21:56:47.143401 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.143370    2583 scope.go:117] "RemoveContainer" containerID="d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"
Jul 30 21:56:47.149949 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.149927    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:56:47.154235 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.154216    2583 scope.go:117] "RemoveContainer" containerID="d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"
Jul 30 21:56:47.154552 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:47.154526    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f\": container with ID starting with d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f not found: ID does not exist" containerID="d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"
Jul 30 21:56:47.154657 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.154576    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f"} err="failed to get container status \"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f\": rpc error: code = NotFound desc = could not find container \"d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f\": container with ID starting with d61a88a1cbb0e9134de983bb5005454c7ff829f0d8a50015c1a7af9f75ead98f not found: ID does not exist"
Jul 30 21:56:47.155661 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.155642    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-2abe52-567858c86b-sw55q"]
Jul 30 21:56:47.164751 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.164730    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:56:47.170374 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.170354    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-2abe52-7f698b4458-gvjrj"]
Jul 30 21:56:47.185246 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.185219    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:56:47.188938 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:47.188917    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-2abe52-749dd59796-c5vnh"]
Jul 30 21:56:48.776890 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:48.776860    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e718140-09a0-464d-bed5-c569885a92c2" path="/var/lib/kubelet/pods/4e718140-09a0-464d-bed5-c569885a92c2/volumes"
Jul 30 21:56:48.777275 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:48.777201    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a779da23-21f4-4689-a35a-d32bcf97642a" path="/var/lib/kubelet/pods/a779da23-21f4-4689-a35a-d32bcf97642a/volumes"
Jul 30 21:56:48.777582 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:48.777553    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" path="/var/lib/kubelet/pods/c8c1718b-86d1-4f41-8ad3-dc8b1a092994/volumes"
Jul 30 21:56:52.984682 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:52.984622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:53.085941 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.085901    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:53.283047 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.282965    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:53.517350 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.517305    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:56:53.638124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638030    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"]
Jul 30 21:56:53.638389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638375    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a779da23-21f4-4689-a35a-d32bcf97642a" containerName="payload-processing"
Jul 30 21:56:53.638389 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638389    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a779da23-21f4-4689-a35a-d32bcf97642a" containerName="payload-processing"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638405    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" containerName="maas-api"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638411    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" containerName="maas-api"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638419    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbd8c30-44a5-4924-8d1f-9a41cad7d424" containerName="revoke-keys"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638424    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbd8c30-44a5-4924-8d1f-9a41cad7d424" containerName="revoke-keys"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638434    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e718140-09a0-464d-bed5-c569885a92c2" containerName="payload-pre-processing"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638439    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e718140-09a0-464d-bed5-c569885a92c2" containerName="payload-pre-processing"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638489    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c1718b-86d1-4f41-8ad3-dc8b1a092994" containerName="maas-api"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638497    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e718140-09a0-464d-bed5-c569885a92c2" containerName="payload-pre-processing"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638503    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a779da23-21f4-4689-a35a-d32bcf97642a" containerName="payload-processing"
Jul 30 21:56:53.638507 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.638509    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbd8c30-44a5-4924-8d1f-9a41cad7d424" containerName="revoke-keys"
Jul 30 21:56:53.641447 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.641425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:53.643044 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.643022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pwwf4\""
Jul 30 21:56:53.649188 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.649168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"]
Jul 30 21:56:53.768106 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.768070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt\" (UID: \"b63eed6e-04e7-4967-8755-2a1121721fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:53.869002 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.868952    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt\" (UID: \"b63eed6e-04e7-4967-8755-2a1121721fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:53.869660 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.869637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt\" (UID: \"b63eed6e-04e7-4967-8755-2a1121721fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:53.954007 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:53.953923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:54.092226 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:54.092014    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"]
Jul 30 21:56:54.094758 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:56:54.094729    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63eed6e_04e7_4967_8755_2a1121721fef.slice/crio-4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb WatchSource:0}: Error finding container 4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb: Status 404 returned error can't find the container with id 4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb
Jul 30 21:56:54.171160 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:54.171119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt" event={"ID":"b63eed6e-04e7-4967-8755-2a1121721fef","Type":"ContainerStarted","Data":"4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb"}
Jul 30 21:56:54.815991 ip-10-0-143-139 kubenswrapper[2583]: E0730 21:56:54.815958    2583 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63eed6e_04e7_4967_8755_2a1121721fef.slice/crio-70b3248ba05e5cb2694d60d3bfda8ec602c1666ed589e0fc136f7594a9c79a26.scope\": RecentStats: unable to find data in memory cache]"
Jul 30 21:56:55.178625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:55.178502    2583 generic.go:356] "Generic (PLEG): container finished" podID="b63eed6e-04e7-4967-8755-2a1121721fef" containerID="70b3248ba05e5cb2694d60d3bfda8ec602c1666ed589e0fc136f7594a9c79a26" exitCode=0
Jul 30 21:56:55.178625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:55.178555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt" event={"ID":"b63eed6e-04e7-4967-8755-2a1121721fef","Type":"ContainerDied","Data":"70b3248ba05e5cb2694d60d3bfda8ec602c1666ed589e0fc136f7594a9c79a26"}
Jul 30 21:56:56.324409 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:56.324382    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:56:56.388356 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:56.388317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca\") pod \"b63eed6e-04e7-4967-8755-2a1121721fef\" (UID: \"b63eed6e-04e7-4967-8755-2a1121721fef\") "
Jul 30 21:56:56.388730 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:56.388708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b63eed6e-04e7-4967-8755-2a1121721fef" (UID: "b63eed6e-04e7-4967-8755-2a1121721fef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:56:56.489809 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:56.489780    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b63eed6e-04e7-4967-8755-2a1121721fef-maas-api-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:56:57.196230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:57.196134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt" event={"ID":"b63eed6e-04e7-4967-8755-2a1121721fef","Type":"ContainerDied","Data":"4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb"}
Jul 30 21:56:57.196230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:57.196193    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fce3dcb0bed44cdabebc6233356cd33300370b7204c9c2aeeb791bcb64efefb"
Jul 30 21:56:57.196230 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:56:57.196151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"
Jul 30 21:57:04.047041 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.046959    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:57:04.047461 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.047205    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" podUID="5849ede3-621f-402e-b41d-7755c5e216c5" containerName="maas-api" containerID="cri-o://103a5912cb379189bd8f0e34fe48f9fd117174ce466ab698accf925e89d92c12" gracePeriod=30
Jul 30 21:57:04.075496 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.075466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:04.078897 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.078869    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"]
Jul 30 21:57:04.081287 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.081261    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-592c1e-29757465-dqcsj"]
Jul 30 21:57:04.109408 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.109370    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:57:04.109635 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.109615    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" podUID="a76b0713-31b1-4395-88cf-4c73e7f3a951" containerName="payload-processing" containerID="cri-o://32ddb4b57356ec002d9ea3ff80f879ed3e499a94d869be3e15568159dbd685ad" gracePeriod=30
Jul 30 21:57:04.116183 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.116044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:57:04.116343 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.116271    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" podUID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" containerName="payload-pre-processing" containerID="cri-o://d72961df84650eb365381a175d3530dcc0ece50370b46f6806e5a1b371cf3a23" gracePeriod=30
Jul 30 21:57:04.148055 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.148010    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:04.200633 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.200557    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:04.237813 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.237780    2583 generic.go:356] "Generic (PLEG): container finished" podID="a76b0713-31b1-4395-88cf-4c73e7f3a951" containerID="32ddb4b57356ec002d9ea3ff80f879ed3e499a94d869be3e15568159dbd685ad" exitCode=0
Jul 30 21:57:04.237952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.237864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" event={"ID":"a76b0713-31b1-4395-88cf-4c73e7f3a951","Type":"ContainerDied","Data":"32ddb4b57356ec002d9ea3ff80f879ed3e499a94d869be3e15568159dbd685ad"}
Jul 30 21:57:04.240193 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.240167    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" containerID="d72961df84650eb365381a175d3530dcc0ece50370b46f6806e5a1b371cf3a23" exitCode=0
Jul 30 21:57:04.240342 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.240236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" event={"ID":"ca1741a4-293b-405d-b9c8-6ceb85391cfe","Type":"ContainerDied","Data":"d72961df84650eb365381a175d3530dcc0ece50370b46f6806e5a1b371cf3a23"}
Jul 30 21:57:04.242338 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.242311    2583 generic.go:356] "Generic (PLEG): container finished" podID="5849ede3-621f-402e-b41d-7755c5e216c5" containerID="103a5912cb379189bd8f0e34fe48f9fd117174ce466ab698accf925e89d92c12" exitCode=0
Jul 30 21:57:04.242450 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.242376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" event={"ID":"5849ede3-621f-402e-b41d-7755c5e216c5","Type":"ContainerDied","Data":"103a5912cb379189bd8f0e34fe48f9fd117174ce466ab698accf925e89d92c12"}
Jul 30 21:57:04.254602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.254555    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"]
Jul 30 21:57:04.256135 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.256107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:04.257101 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.257072    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-592c1e-gqzpt"]
Jul 30 21:57:04.379148 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.379123    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:57:04.413217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.413194    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:57:04.420040 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.420014    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:57:04.454399 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.454367    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca\") pod \"5849ede3-621f-402e-b41d-7755c5e216c5\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") "
Jul 30 21:57:04.454632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.454414    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls\") pod \"5849ede3-621f-402e-b41d-7755c5e216c5\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") "
Jul 30 21:57:04.454632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.454502    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9f8d\" (UniqueName: \"kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d\") pod \"5849ede3-621f-402e-b41d-7755c5e216c5\" (UID: \"5849ede3-621f-402e-b41d-7755c5e216c5\") "
Jul 30 21:57:04.454811 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.454709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5849ede3-621f-402e-b41d-7755c5e216c5" (UID: "5849ede3-621f-402e-b41d-7755c5e216c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:57:04.456602 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.456548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5849ede3-621f-402e-b41d-7755c5e216c5" (UID: "5849ede3-621f-402e-b41d-7755c5e216c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 30 21:57:04.456723 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.456643    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d" (OuterVolumeSpecName: "kube-api-access-d9f8d") pod "5849ede3-621f-402e-b41d-7755c5e216c5" (UID: "5849ede3-621f-402e-b41d-7755c5e216c5"). InnerVolumeSpecName "kube-api-access-d9f8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:57:04.555349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555316    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume\") pod \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") "
Jul 30 21:57:04.555349 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzw49\" (UniqueName: \"kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49\") pod \"a76b0713-31b1-4395-88cf-4c73e7f3a951\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") "
Jul 30 21:57:04.555600 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555400    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgvdk\" (UniqueName: \"kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk\") pod \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\" (UID: \"ca1741a4-293b-405d-b9c8-6ceb85391cfe\") "
Jul 30 21:57:04.555600 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555425    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume\") pod \"a76b0713-31b1-4395-88cf-4c73e7f3a951\" (UID: \"a76b0713-31b1-4395-88cf-4c73e7f3a951\") "
Jul 30 21:57:04.555712 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555693    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5849ede3-621f-402e-b41d-7755c5e216c5-maas-api-tls\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.555750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555720    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9f8d\" (UniqueName: \"kubernetes.io/projected/5849ede3-621f-402e-b41d-7755c5e216c5-kube-api-access-d9f8d\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.555750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555736    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5849ede3-621f-402e-b41d-7755c5e216c5-openshift-service-ca\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.555903 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555873    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca1741a4-293b-405d-b9c8-6ceb85391cfe" (UID: "ca1741a4-293b-405d-b9c8-6ceb85391cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:57:04.555941 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.555911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a76b0713-31b1-4395-88cf-4c73e7f3a951" (UID: "a76b0713-31b1-4395-88cf-4c73e7f3a951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 30 21:57:04.557695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.557673    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49" (OuterVolumeSpecName: "kube-api-access-tzw49") pod "a76b0713-31b1-4395-88cf-4c73e7f3a951" (UID: "a76b0713-31b1-4395-88cf-4c73e7f3a951"). InnerVolumeSpecName "kube-api-access-tzw49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:57:04.557752 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.557688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk" (OuterVolumeSpecName: "kube-api-access-xgvdk") pod "ca1741a4-293b-405d-b9c8-6ceb85391cfe" (UID: "ca1741a4-293b-405d-b9c8-6ceb85391cfe"). InnerVolumeSpecName "kube-api-access-xgvdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 30 21:57:04.640683 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.640585    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:04.656667 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.656634    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvdk\" (UniqueName: \"kubernetes.io/projected/ca1741a4-293b-405d-b9c8-6ceb85391cfe-kube-api-access-xgvdk\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.656667 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.656660    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a76b0713-31b1-4395-88cf-4c73e7f3a951-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.656667 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.656669    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1741a4-293b-405d-b9c8-6ceb85391cfe-plugins-config-volume\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.656886 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.656677    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzw49\" (UniqueName: \"kubernetes.io/projected/a76b0713-31b1-4395-88cf-4c73e7f3a951-kube-api-access-tzw49\") on node \"ip-10-0-143-139.ec2.internal\" DevicePath \"\""
Jul 30 21:57:04.777071 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.777034    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ecd983-798d-41c4-82cf-942f23ba10b4" path="/var/lib/kubelet/pods/41ecd983-798d-41c4-82cf-942f23ba10b4/volumes"
Jul 30 21:57:04.777321 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:04.777308    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b63eed6e-04e7-4967-8755-2a1121721fef" path="/var/lib/kubelet/pods/b63eed6e-04e7-4967-8755-2a1121721fef/volumes"
Jul 30 21:57:05.248639 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.248608    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"
Jul 30 21:57:05.248639 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.248622    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7" event={"ID":"a76b0713-31b1-4395-88cf-4c73e7f3a951","Type":"ContainerDied","Data":"9ff79b4985e3a93752f87731b7fadf857277a24c1af88d53dd612d93caa5cc44"}
Jul 30 21:57:05.249162 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.248696    2583 scope.go:117] "RemoveContainer" containerID="32ddb4b57356ec002d9ea3ff80f879ed3e499a94d869be3e15568159dbd685ad"
Jul 30 21:57:05.250170 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.250037    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"
Jul 30 21:57:05.250170 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.250110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg" event={"ID":"ca1741a4-293b-405d-b9c8-6ceb85391cfe","Type":"ContainerDied","Data":"8add70e4a814b1240c8ab78a129ed8ff5224bb2ccbcb989927fc9dace2a0c196"}
Jul 30 21:57:05.251679 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.251650    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"
Jul 30 21:57:05.251793 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.251695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd" event={"ID":"5849ede3-621f-402e-b41d-7755c5e216c5","Type":"ContainerDied","Data":"a25c96a57263d58d1ef4032f80f9913e06896ee810afefd53f50ae1a9e25484f"}
Jul 30 21:57:05.258765 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.258737    2583 scope.go:117] "RemoveContainer" containerID="d72961df84650eb365381a175d3530dcc0ece50370b46f6806e5a1b371cf3a23"
Jul 30 21:57:05.266841 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.266820    2583 scope.go:117] "RemoveContainer" containerID="103a5912cb379189bd8f0e34fe48f9fd117174ce466ab698accf925e89d92c12"
Jul 30 21:57:05.267068 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.267052    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:57:05.271640 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.271620    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-592c1e-67755885c-rnprd"]
Jul 30 21:57:05.279538 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.279514    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:57:05.284108 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.284084    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-592c1e-6748576dc6-q4bsg"]
Jul 30 21:57:05.295750 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.295729    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:57:05.299241 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:05.299220    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-592c1e-796547bb66-j86t7"]
Jul 30 21:57:06.777335 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:06.777300    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5849ede3-621f-402e-b41d-7755c5e216c5" path="/var/lib/kubelet/pods/5849ede3-621f-402e-b41d-7755c5e216c5/volumes"
Jul 30 21:57:06.777771 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:06.777659    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76b0713-31b1-4395-88cf-4c73e7f3a951" path="/var/lib/kubelet/pods/a76b0713-31b1-4395-88cf-4c73e7f3a951/volumes"
Jul 30 21:57:06.777952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:06.777939    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" path="/var/lib/kubelet/pods/ca1741a4-293b-405d-b9c8-6ceb85391cfe/volumes"
Jul 30 21:57:09.290536 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:09.290511    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:57:09.298388 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:09.298365    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6n8x_de3a92fe-ffbc-4438-9801-81c920293a4f/ovn-acl-logging/0.log"
Jul 30 21:57:09.582632 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:09.582524    2583 scope.go:117] "RemoveContainer" containerID="7b388f146913317c88234e1a062c62c521c967101cac84d7acb3046df5325f9e"
Jul 30 21:57:10.452252 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:10.452215    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:10.551050 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:10.551012    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:10.729545 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:10.729507    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:10.974600 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:10.974519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:11.464147 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:11.464108    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b445799db-4z7zm_a742d93b-79cd-4e90-8d9d-c343ac9fd72c/authorino/0.log"
Jul 30 21:57:16.964718 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:16.964687    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-clsg8_5090fc76-a383-4e9e-8a71-719d87ec90bd/manager/0.log"
Jul 30 21:57:17.088198 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.088172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-876g6_83e40ff0-3ce8-4f14-a555-364c465aa18a/manager/0.log"
Jul 30 21:57:17.224662 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.224642    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7c48cdc9-zqkmv_492f2e3f-3e4a-401f-aa2e-c40ee18b05f9/manager/0.log"
Jul 30 21:57:17.361978 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.361947    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s4zw5_dc3dfbc2-af6b-4bb6-ad41-3542189d2859/server/0.log"
Jul 30 21:57:17.490415 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.490334    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dtlkp_f3349474-912f-4c9b-bd42-4cda739d3231/manager/0.log"
Jul 30 21:57:17.599560 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.599534    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcd94768-fdhfj_1543b89f-afdb-430f-9be1-ac2b362e2ee7/copy-manifests/0.log"
Jul 30 21:57:17.617288 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:17.617251    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcd94768-fdhfj_1543b89f-afdb-430f-9be1-ac2b362e2ee7/manager/0.log"
Jul 30 21:57:18.355647 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:18.355614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757435-m28n2_0e86f508-6a66-4207-a4e0-0fb7e0283c60/cleanup/0.log"
Jul 30 21:57:18.475313 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:18.475280    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29757450-sptcb_8908ed36-f259-4334-8caa-abcb1c18db9f/cleanup/0.log"
Jul 30 21:57:18.713032 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:18.712940    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kgft4_19e75ea8-7888-4b10-ad02-2b78c142e95a/postgres/0.log"
Jul 30 21:57:19.597607 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:19.597556    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b445799db-4z7zm_a742d93b-79cd-4e90-8d9d-c343ac9fd72c/authorino/0.log"
Jul 30 21:57:19.712611 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:19.712583    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zfdnb_1220c293-f6db-4875-8403-3536ad63e23e/manager/0.log"
Jul 30 21:57:19.834856 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:19.834820    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-dzj2j_82d57060-bfa9-485e-9045-df0a1e7256d0/manager/0.log"
Jul 30 21:57:20.081124 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:20.081091    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-8r25m_2267cd44-8396-4f33-868f-0e835620741f/manager/0.log"
Jul 30 21:57:20.328747 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:20.328711    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-hmwjr_2797eed7-af70-476a-9ce4-1e50a70c0e30/manager/0.log"
Jul 30 21:57:21.038620 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:21.038555    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65ffdd497f-9dksv_a79443e1-d3bc-48dc-89b1-942ea2625008/kube-auth-proxy/0.log"
Jul 30 21:57:21.406786 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:21.406693    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:57:21.921876 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:21.921847    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mgqpt_228a93d3-ca8a-4f74-9494-4ac0c2808531/main/0.log"
Jul 30 21:57:28.132014 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.131977    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ptt2n/must-gather-9jtcm"]
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132300    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" containerName="payload-pre-processing"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132312    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" containerName="payload-pre-processing"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132325    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5849ede3-621f-402e-b41d-7755c5e216c5" containerName="maas-api"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132331    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5849ede3-621f-402e-b41d-7755c5e216c5" containerName="maas-api"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132341    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b63eed6e-04e7-4967-8755-2a1121721fef" containerName="revoke-keys"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b63eed6e-04e7-4967-8755-2a1121721fef" containerName="revoke-keys"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132358    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76b0713-31b1-4395-88cf-4c73e7f3a951" containerName="payload-processing"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132363    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76b0713-31b1-4395-88cf-4c73e7f3a951" containerName="payload-processing"
Jul 30 21:57:28.132414 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132416    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1741a4-293b-405d-b9c8-6ceb85391cfe" containerName="payload-pre-processing"
Jul 30 21:57:28.132778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132426    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5849ede3-621f-402e-b41d-7755c5e216c5" containerName="maas-api"
Jul 30 21:57:28.132778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132432    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76b0713-31b1-4395-88cf-4c73e7f3a951" containerName="payload-processing"
Jul 30 21:57:28.132778 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.132438    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b63eed6e-04e7-4967-8755-2a1121721fef" containerName="revoke-keys"
Jul 30 21:57:28.135517 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.135494    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.137767 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.137742    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ptt2n\"/\"openshift-service-ca.crt\""
Jul 30 21:57:28.137891 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.137854    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ptt2n\"/\"default-dockercfg-qrlrh\""
Jul 30 21:57:28.138062 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.138041    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ptt2n\"/\"kube-root-ca.crt\""
Jul 30 21:57:28.138881 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.138859    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:28.145721 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.145697    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptt2n/must-gather-9jtcm"]
Jul 30 21:57:28.159320 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.159297    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6wm\" (UniqueName: \"kubernetes.io/projected/72993efd-5c99-4ad3-81bb-66688c2c8b27-kube-api-access-lz6wm\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.159446 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.159426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72993efd-5c99-4ad3-81bb-66688c2c8b27-must-gather-output\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.260292 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.260253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72993efd-5c99-4ad3-81bb-66688c2c8b27-must-gather-output\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.260292 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.260297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6wm\" (UniqueName: \"kubernetes.io/projected/72993efd-5c99-4ad3-81bb-66688c2c8b27-kube-api-access-lz6wm\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.260618 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.260598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/72993efd-5c99-4ad3-81bb-66688c2c8b27-must-gather-output\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.268132 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.268109    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6wm\" (UniqueName: \"kubernetes.io/projected/72993efd-5c99-4ad3-81bb-66688c2c8b27-kube-api-access-lz6wm\") pod \"must-gather-9jtcm\" (UID: \"72993efd-5c99-4ad3-81bb-66688c2c8b27\") " pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.445954 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.445858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptt2n/must-gather-9jtcm"
Jul 30 21:57:28.577593 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.577547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptt2n/must-gather-9jtcm"]
Jul 30 21:57:28.580445 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:57:28.580415    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72993efd_5c99_4ad3_81bb_66688c2c8b27.slice/crio-20fad9c457b46ae6494d002efeb137e60362e137427a022af2dfce87645e2274 WatchSource:0}: Error finding container 20fad9c457b46ae6494d002efeb137e60362e137427a022af2dfce87645e2274: Status 404 returned error can't find the container with id 20fad9c457b46ae6494d002efeb137e60362e137427a022af2dfce87645e2274
Jul 30 21:57:28.582743 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:28.582728    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 30 21:57:29.379988 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:29.379960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptt2n/must-gather-9jtcm" event={"ID":"72993efd-5c99-4ad3-81bb-66688c2c8b27","Type":"ContainerStarted","Data":"20fad9c457b46ae6494d002efeb137e60362e137427a022af2dfce87645e2274"}
Jul 30 21:57:29.502914 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:29.502873    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log" limit=1024
Jul 30 21:57:30.387405 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:30.387358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptt2n/must-gather-9jtcm" event={"ID":"72993efd-5c99-4ad3-81bb-66688c2c8b27","Type":"ContainerStarted","Data":"18361fd34b27c75338243b6568196602d8694d4ec807317c7aeca42bb851f42e"}
Jul 30 21:57:30.387925 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:30.387420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptt2n/must-gather-9jtcm" event={"ID":"72993efd-5c99-4ad3-81bb-66688c2c8b27","Type":"ContainerStarted","Data":"1206c7ab27b23ff7f7de2f14239719f2eb284ae5076c22f44d794ef2ebb87a74"}
Jul 30 21:57:30.402587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:30.402512    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ptt2n/must-gather-9jtcm" podStartSLOduration=1.613112868 podStartE2EDuration="2.402497285s" totalImagesPullingTime="789.384417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 21:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:57:30.402098268 +0000 UTC m=+3622.168060145" watchObservedRunningTime="2026-07-30 21:57:30.402497285 +0000 UTC m=+3622.168459161"
Jul 30 21:57:30.914387 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:30.914357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gts47_dfa27313-a228-4ddc-abc6-78cb031f2476/global-pull-secret-syncer/0.log"
Jul 30 21:57:31.052627 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:31.052593    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sb66j_46b4b672-d044-4552-a556-d09e24baecd5/konnectivity-agent/0.log"
Jul 30 21:57:31.196591 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:31.196488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-139.ec2.internal_2d952314a66eed41f4a061771aac559f/haproxy/0.log"
Jul 30 21:57:36.109625 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:36.108811    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b445799db-4z7zm_a742d93b-79cd-4e90-8d9d-c343ac9fd72c/authorino/0.log"
Jul 30 21:57:36.145523 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:36.145481    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-zfdnb_1220c293-f6db-4875-8403-3536ad63e23e/manager/0.log"
Jul 30 21:57:36.181952 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:36.181915    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-dzj2j_82d57060-bfa9-485e-9045-df0a1e7256d0/manager/0.log"
Jul 30 21:57:36.421635 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:36.419706    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-8r25m_2267cd44-8396-4f33-868f-0e835620741f/manager/0.log"
Jul 30 21:57:36.574844 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:36.574685    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-hmwjr_2797eed7-af70-476a-9ce4-1e50a70c0e30/manager/0.log"
Jul 30 21:57:38.568948 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:38.568908    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4546_d12048cd-30bf-4440-800d-d3b07f30aec2/node-exporter/0.log"
Jul 30 21:57:38.596795 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:38.596731    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4546_d12048cd-30bf-4440-800d-d3b07f30aec2/kube-rbac-proxy/0.log"
Jul 30 21:57:38.651528 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:38.651431    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s4546_d12048cd-30bf-4440-800d-d3b07f30aec2/init-textfile/0.log"
Jul 30 21:57:41.356800 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.356760    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"]
Jul 30 21:57:41.364262 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.364235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.369963 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.369935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"]
Jul 30 21:57:41.499423 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.499380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-lib-modules\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.499606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.499444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxbq\" (UniqueName: \"kubernetes.io/projected/28aae90b-56d1-443c-894d-e4cd315dbba6-kube-api-access-rlxbq\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.499606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.499498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-proc\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.499606 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.499583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-podres\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.499757 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.499626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-sys\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.600950 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.600905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-lib-modules\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.601204 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxbq\" (UniqueName: \"kubernetes.io/projected/28aae90b-56d1-443c-894d-e4cd315dbba6-kube-api-access-rlxbq\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.601333 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-proc\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.601448 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601437    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-podres\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.601584 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-sys\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.601906 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-lib-modules\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.602034 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-sys\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.602120 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-podres\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.602198 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.601825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28aae90b-56d1-443c-894d-e4cd315dbba6-proc\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.618727 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.618659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxbq\" (UniqueName: \"kubernetes.io/projected/28aae90b-56d1-443c-894d-e4cd315dbba6-kube-api-access-rlxbq\") pod \"perf-node-gather-daemonset-59nv7\" (UID: \"28aae90b-56d1-443c-894d-e4cd315dbba6\") " pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.689988 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.689950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:41.893587 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:41.893538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"]
Jul 30 21:57:41.896017 ip-10-0-143-139 kubenswrapper[2583]: W0730 21:57:41.895983    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28aae90b_56d1_443c_894d_e4cd315dbba6.slice/crio-d87d0ea5b6b4ab2eb3fbd913216dada1406b6d299adc79387223242d8d58d6f5 WatchSource:0}: Error finding container d87d0ea5b6b4ab2eb3fbd913216dada1406b6d299adc79387223242d8d58d6f5: Status 404 returned error can't find the container with id d87d0ea5b6b4ab2eb3fbd913216dada1406b6d299adc79387223242d8d58d6f5
Jul 30 21:57:42.486804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:42.486697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7" event={"ID":"28aae90b-56d1-443c-894d-e4cd315dbba6","Type":"ContainerStarted","Data":"fbb881a483d77319016a1689bd25d67f5d04970bc174e1527ff99dce2f343644"}
Jul 30 21:57:42.486804 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:42.486766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7" event={"ID":"28aae90b-56d1-443c-894d-e4cd315dbba6","Type":"ContainerStarted","Data":"d87d0ea5b6b4ab2eb3fbd913216dada1406b6d299adc79387223242d8d58d6f5"}
Jul 30 21:57:42.487324 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:42.487300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:42.505206 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:42.505143    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7" podStartSLOduration=1.505125177 podStartE2EDuration="1.505125177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 21:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 21:57:42.502576571 +0000 UTC m=+3634.268538441" watchObservedRunningTime="2026-07-30 21:57:42.505125177 +0000 UTC m=+3634.271087057"
Jul 30 21:57:43.797191 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:43.797163    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zl2hb_9a5fc1e5-6b71-4156-9cd8-cda500ec1c22/dns/0.log"
Jul 30 21:57:43.818811 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:43.818778    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zl2hb_9a5fc1e5-6b71-4156-9cd8-cda500ec1c22/kube-rbac-proxy/0.log"
Jul 30 21:57:43.894577 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:43.894542    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s2nhn_2a4ce903-51af-4a3b-a1be-ae00c1bccb0b/dns-node-resolver/0.log"
Jul 30 21:57:44.512864 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:44.512817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q5vql_7e57f05e-01a1-43ac-abb3-32f6daed266e/node-ca/0.log"
Jul 30 21:57:45.661626 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:45.661598    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65ffdd497f-9dksv_a79443e1-d3bc-48dc-89b1-942ea2625008/kube-auth-proxy/0.log"
Jul 30 21:57:45.763551 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:45.763495    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:57:46.625855 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:46.625816    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-flp9x_edaefa0a-8e43-40cd-a68b-1bf0a3ee60e2/serve-healthcheck-canary/0.log"
Jul 30 21:57:47.142151 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:47.142119    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f9glr_dcf45e82-3aab-44db-a754-efe881059ce7/kube-rbac-proxy/0.log"
Jul 30 21:57:47.164113 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:47.164072    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f9glr_dcf45e82-3aab-44db-a754-efe881059ce7/exporter/0.log"
Jul 30 21:57:47.188052 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:47.188022    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f9glr_dcf45e82-3aab-44db-a754-efe881059ce7/extractor/0.log"
Jul 30 21:57:49.418770 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:49.418737    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-clsg8_5090fc76-a383-4e9e-8a71-719d87ec90bd/manager/0.log"
Jul 30 21:57:49.514492 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:49.514458    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ptt2n/perf-node-gather-daemonset-59nv7"
Jul 30 21:57:49.877538 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:49.877503    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-876g6_83e40ff0-3ce8-4f14-a555-364c465aa18a/manager/0.log"
Jul 30 21:57:50.197320 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:50.197227    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7d7c48cdc9-zqkmv_492f2e3f-3e4a-401f-aa2e-c40ee18b05f9/manager/0.log"
Jul 30 21:57:50.221336 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:50.221295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-s4zw5_dc3dfbc2-af6b-4bb6-ad41-3542189d2859/server/0.log"
Jul 30 21:57:50.352695 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:50.352647    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-dtlkp_f3349474-912f-4c9b-bd42-4cda739d3231/manager/0.log"
Jul 30 21:57:50.556807 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:50.556769    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcd94768-fdhfj_1543b89f-afdb-430f-9be1-ac2b362e2ee7/manager/0.log"
Jul 30 21:57:50.594282 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:50.594251    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ddcd94768-fdhfj_1543b89f-afdb-430f-9be1-ac2b362e2ee7/copy-manifests/0.log"
Jul 30 21:57:52.431741 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:52.431701    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65ffdd497f-9dksv_a79443e1-d3bc-48dc-89b1-942ea2625008/kube-auth-proxy/0.log"
Jul 30 21:57:52.522100 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:52.522062    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:57:54.109217 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:54.109172    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65ffdd497f-9dksv_a79443e1-d3bc-48dc-89b1-942ea2625008/kube-auth-proxy/0.log"
Jul 30 21:57:54.192329 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:54.192296    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-stmd5_82fa765e-1635-4898-afdb-718f76ff6b11/payload-processing/0.log"
Jul 30 21:57:58.770604 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.770579    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/kube-multus-additional-cni-plugins/0.log"
Jul 30 21:57:58.792972 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.792942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/egress-router-binary-copy/0.log"
Jul 30 21:57:58.815138 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.815109    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/cni-plugins/0.log"
Jul 30 21:57:58.836781 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.836756    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/bond-cni-plugin/0.log"
Jul 30 21:57:58.857418 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.857395    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/routeoverride-cni/0.log"
Jul 30 21:57:58.879208 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.879180    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/whereabouts-cni-bincopy/0.log"
Jul 30 21:57:58.906726 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:58.906692    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9ftb6_ffa0da2f-098a-4e66-9d21-78c92c2b5b71/whereabouts-cni/0.log"
Jul 30 21:57:59.285699 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:59.285665    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r8txp_8982e187-c64c-44c8-8047-7c7b532aa65d/kube-multus/0.log"
Jul 30 21:57:59.389307 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:59.389276    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nlj6v_8a763470-0568-4720-a87c-636745df6053/network-metrics-daemon/0.log"
Jul 30 21:57:59.406796 ip-10-0-143-139 kubenswrapper[2583]: I0730 21:57:59.406759    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nlj6v_8a763470-0568-4720-a87c-636745df6053/kube-rbac-proxy/0.log"