Sep 07 10:18:28.149734 ip-10-0-142-36 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 07 10:18:28.149746 ip-10-0-142-36 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 07 10:18:28.149754 ip-10-0-142-36 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 07 10:18:28.150052 ip-10-0-142-36 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 07 10:18:38.319692 ip-10-0-142-36 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 07 10:18:38.319709 ip-10-0-142-36 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 28ef767ae8b14096b926e6209c38f74d --
Sep 07 10:20:11.874559 ip-10-0-142-36 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 10:20:12.297967 ip-10-0-142-36 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:20:12.297967 ip-10-0-142-36 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 10:20:12.297967 ip-10-0-142-36 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:20:12.297967 ip-10-0-142-36 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 10:20:12.297967 ip-10-0-142-36 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:20:12.299828 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.299747    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 10:20:12.303752 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303738    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.303752 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303752    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303756    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303759    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303761    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303765    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303768    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303770    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303773    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303775    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303780    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303783    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303787    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303790    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303793    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303795    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303798    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303800    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303803    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303806    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303808    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.303813 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303811    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303813    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303816    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303818    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303821    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303824    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303826    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303829    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303832    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303834    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303837    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303839    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303841    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303844    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303846    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303848    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303851    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303853    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303856    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.304265 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303858    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303861    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303863    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303866    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303870    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303874    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303878    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303882    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303884    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303886    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303889    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303891    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303894    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303896    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303899    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303901    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303904    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303906    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303908    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.304761 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303911    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303913    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303916    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303918    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303920    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303923    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303925    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303928    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303931    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303933    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303936    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303939    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303942    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303944    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303946    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303948    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303951    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303954    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303957    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303959    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.305231 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303962    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303964    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303966    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303969    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303971    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303974    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303976    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303979    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303982    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303984    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303987    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303989    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303992    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303994    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303997    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.303999    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304001    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304004    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304006    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304008    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.306012 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304667    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304673    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304676    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304679    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304681    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304684    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304686    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304689    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304691    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304694    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304697    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304699    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304702    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304704    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304707    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304709    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304712    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304714    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304716    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304719    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.306633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304723    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304726    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304729    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304731    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304734    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304736    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304739    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304741    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304744    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304746    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304749    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304751    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304753    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304756    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304758    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304760    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304763    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304765    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304768    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.307124 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304770    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304773    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304775    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304779    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304783    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304786    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304789    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304791    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304794    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304797    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304799    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304802    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304806    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304809    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304812    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304815    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304817    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304820    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304822    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.307771 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304824    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304827    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304829    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304832    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304834    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304836    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304838    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304840    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304843    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304846    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304848    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304851    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304854    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304856    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304859    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304861    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304863    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304866    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304869    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304873    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.308259 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304875    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304878    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304880    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304882    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304887    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304889    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304892    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304895    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304897    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304900    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304903    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304905    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304907    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304910    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304912    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304915    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304917    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304919    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304922    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304924    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.308760 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.304926    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307769    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307777    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307781    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307784    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307786    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307789    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307792    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307794    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307797    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307800    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307802    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307805    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307807    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307810    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307813    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307816    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307819    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307821    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307824    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.309228 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307827    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307830    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307832    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307835    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307838    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307843    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307846    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307849    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307851    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307854    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307856    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307859    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307863    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307867    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307870    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307873    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307876    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307879    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307881    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307884    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.309724 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307887    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307889    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307892    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307895    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307897    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307900    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307903    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307905    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307909    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307911    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307914    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307917    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307920    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307923    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307925    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307928    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307930    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307933    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307935    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307937    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.310185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307940    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307942    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307944    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307947    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307949    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307951    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307955    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307959    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307962    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307964    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307967    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307969    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307972    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307974    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307977    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307979    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307982    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307984    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307987    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307989    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.310677 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307992    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307995    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.307997    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308000    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308005    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308008    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308011    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308013    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308016    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308018    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308021    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308023    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308026    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308029    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308031    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308034    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308036    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308039    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308042    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308044    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.311152 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308104    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308135    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308142    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308146    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308150    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308153    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308157    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308161    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308164    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308167    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308170    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308173    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308177    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308180    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308183    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308185    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308188    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308191    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308196    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308198    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308201    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308204    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308207    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308210    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 10:20:12.311653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308217    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308220    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308223    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308226    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308229    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308231    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308234    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308237    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308241    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308243    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308246    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308249    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308252    2562 flags.go:64] FLAG: --enable-server="true"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308254    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308258    2562 flags.go:64] FLAG: --event-burst="100"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308262    2562 flags.go:64] FLAG: --event-qps="50"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308264    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308268    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308271    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308274    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308277    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308280    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308283    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308286    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308288    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 10:20:12.312199 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308291    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308294    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308313    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308316    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308319    2562 flags.go:64] FLAG: --feature-gates=""
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308325    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308328    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308331    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308334    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308336    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308339    2562 flags.go:64] FLAG: --help="false"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308342    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308345    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308348    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308351    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308354    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308357    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308360    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308363    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308365    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308368    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308371    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308374    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308377    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 07 10:20:12.312859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308379    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308382    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308385    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308388    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308390    2562 flags.go:64] FLAG: --lock-file=""
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308393    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308396    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308398    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308404    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308406    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308410    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308413    2562 flags.go:64] FLAG: --logging-format="text"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308416    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308419    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308422    2562 flags.go:64] FLAG: --manifest-url=""
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308424    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308429    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308431    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308435    2562 flags.go:64] FLAG: --max-pods="110"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308438    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308441    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308444    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308447    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308449    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308452    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 10:20:12.313439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308457    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308464    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308467    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308469    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308472    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308475    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308481    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308483    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308486    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308503    2562 flags.go:64] FLAG: --port="10250"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308507    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308510    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-047a52804420f4fcc"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308514    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308517    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308520    2562 flags.go:64] FLAG: --register-node="true"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308522    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308526    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308528    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308532    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308535    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308539    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308542    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308545    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308547    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308550    2562 flags.go:64] FLAG: --runonce="false"
Sep 07 10:20:12.314025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308553    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308556    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308559    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308562    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308565    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308567    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308570    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308573    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308578    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308580    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308583    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308586    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308589    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308591    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308594    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308599    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308602    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308605    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308609    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308611    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308614    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308617    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308620    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308623    2562 flags.go:64] FLAG: --v="2"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308626    2562 flags.go:64] FLAG: --version="false"
Sep 07 10:20:12.314634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308630    2562 flags.go:64] FLAG: --vmodule=""
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308634    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.308637    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308743    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308747    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308750    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308753    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308755    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308758    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308762    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308764    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308767    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308770    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308772    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308775    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308779    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308781    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308784    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308786    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308789    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.315409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308791    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308794    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308798    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308801    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308803    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308806    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308808    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308811    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308813    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308816    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308818    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308821    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308823    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308825    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308828    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308831    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308833    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308836    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308838    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.315876 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308841    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308843    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308845    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308848    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308850    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308853    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308855    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308857    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308861    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308864    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308866    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308868    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308871    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308873    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308876    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308880    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308884    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308887    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308890    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308892    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.316334 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308895    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308897    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308899    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308902    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308905    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308908    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308911    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308913    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308916    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308918    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308921    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308923    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308926    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308928    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308931    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308933    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308936    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308938    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308941    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308943    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.316915 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308947    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308949    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308952    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308955    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308957    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308959    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308962    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308966    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308970    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308973    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308976    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308979    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308981    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308984    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308987    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308989    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308992    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308994    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.308997    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.309000    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.317415 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.309002    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.317893 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.309005    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.317893 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.309008    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.317893 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.309752    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:20:12.318116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.318098    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 10:20:12.318144 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.318116    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 10:20:12.319834 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.319825    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 10:20:12.319870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.319835    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 10:20:12.319898 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319889    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.319898 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319894    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.319898 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319898    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319901    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319904    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319907    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319910    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319912    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319915    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319918    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319920    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319923    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319925    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319928    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319930    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319933    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319936    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319938    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319942    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319946    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319949    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319951    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.319976 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319954    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319956    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319958    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319961    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319964    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319966    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319969    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319972    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319975    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319977    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319979    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319982    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319985    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319988    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319991    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319993    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319995    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.319998    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320000    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320003    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.320467 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320005    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320008    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320010    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320013    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320015    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320017    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320020    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320022    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320025    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320027    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320030    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320032    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320034    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320038    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320042    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320045    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320049    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320052    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320054    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320057    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.320944 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320061    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320063    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320066    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320069    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320071    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320074    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320077    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320079    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320082    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320084    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320086    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320090    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320094    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320097    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320100    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320102    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320105    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320107    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320110    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.321474 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320112    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320114    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320117    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320119    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320122    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320124    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320127    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320130    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320133    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320135    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320138    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320140    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320143    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320145    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320148    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320151    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320153    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.321947 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320156    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.320160    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320265    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320269    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320272    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320274    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320277    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320279    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320282    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320284    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320287    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320290    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320292    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320295    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320312    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:20:12.322353 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320316    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320320    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320322    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320325    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320327    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320330    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320332    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320335    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320338    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320341    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320343    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320347    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320351    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320354    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320356    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320360    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320363    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320366    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320368    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320370    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:20:12.322708 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320373    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320375    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320377    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320380    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320382    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320385    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320387    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320390    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320392    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320394    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320397    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320399    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320402    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320405    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320407    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320410    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320412    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320415    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320417    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320420    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:20:12.323184 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320422    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320425    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320427    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320430    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320432    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320435    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320437    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320439    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320443    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320445    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320448    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320451    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320454    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320456    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320458    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320461    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320463    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320466    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320468    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320471    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:20:12.323656 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320473    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320476    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320479    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320481    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320484    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320486    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320489    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320491    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320493    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320496    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320498    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320501    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320503    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320506    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320508    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320510    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320513    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320515    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320517    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320520    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:20:12.324148 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320522    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320525    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320527    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320530    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320532    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:12.320536    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.320540    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.320640    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 10:20:12.324633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.322786    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 10:20:12.324832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.324783    2562 server.go:1019] "Starting client certificate rotation"
Sep 07 10:20:12.324898 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.324880    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:20:12.324932 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.324923    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:20:12.327530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.327517    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:20:12.328609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.328590    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:20:12.339538 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.339516    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 07 10:20:12.343157 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.343138    2562 log.go:25] "Validated CRI v1 image API"
Sep 07 10:20:12.344008 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.343991    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 10:20:12.347387 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.347368    2562 fs.go:135] Filesystem UUIDs: map[6a3a3858-0390-4f22-9729-da7dd18094c5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 919df864-dcca-4b64-9e7c-4f2e526a9799:/dev/nvme0n1p3]
Sep 07 10:20:12.347445 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.347389    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 10:20:12.353591 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.353488    2562 manager.go:216] Machine: {Timestamp:2026-09-07 10:20:12.352088581 +0000 UTC m=+0.362741885 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3196512 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21be6b859cca88168bca8a5c0d0360 SystemUUID:ec21be6b-859c-ca88-168b-ca8a5c0d0360 BootID:28ef767a-e8b1-4096-b926-e6209c38f74d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7d:44:25:a8:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7d:44:25:a8:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:6a:70:b6:23:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 10:20:12.353591 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.353589    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 10:20:12.353688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.353665    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 10:20:12.356152 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.356136    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:20:12.357551 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.357526    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 10:20:12.357684 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.357555    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 10:20:12.358479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.358468    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 10:20:12.358525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.358481    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 10:20:12.358525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.358493    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:20:12.359110 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.359100    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:20:12.360849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.360840    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 10:20:12.366092 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366081    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:20:12.366192 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366182    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 10:20:12.366222 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366211    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 10:20:12.366251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366227    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 10:20:12.366251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366247    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 10:20:12.366312 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366259    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 07 10:20:12.366312 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366271    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 10:20:12.366610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366599    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:20:12.366654 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.366617    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:20:12.369181 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.369164    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 10:20:12.372166 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372151    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 10:20:12.372241 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372189    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 10:20:12.372389 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372377    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 10:20:12.372453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372396    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 10:20:12.372453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372406    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 10:20:12.372453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372415    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 10:20:12.372453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372426    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 10:20:12.372453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.372434    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 10:20:12.373012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373002    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 10:20:12.373059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373014    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 10:20:12.373059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373025    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 10:20:12.373059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373034    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 10:20:12.373059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373046    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 10:20:12.373059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.373057    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 10:20:12.374997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.374986    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 10:20:12.375059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.374999    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 10:20:12.377062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.377043    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkg88"
Sep 07 10:20:12.377984 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.377971    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 10:20:12.378778 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.378756    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 10:20:12.378778 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.378759    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 10:20:12.379097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.379085    2562 server.go:1268] "Started kubelet"
Sep 07 10:20:12.379208 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.379160    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 10:20:12.379822 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.379720    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 10:20:12.379822 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.379796    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 10:20:12.380236 ip-10-0-142-36 systemd[1]: Started Kubernetes Kubelet.
Sep 07 10:20:12.381054 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.381033    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 10:20:12.382597 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.382571    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 10:20:12.383853 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.383827    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 10:20:12.384653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.384634    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkg88"
Sep 07 10:20:12.388986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.388967    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 10:20:12.388986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.388980    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 10:20:12.389125 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.389041    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.389723 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.389703    2562 factory.go:55] Registering systemd factory
Sep 07 10:20:12.389723 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.389724    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 07 10:20:12.389887 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.389807    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 10:20:12.389887 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.389878    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 10:20:12.389887 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.389890    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 10:20:12.390059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390002    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 10:20:12.390059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390010    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 10:20:12.390059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390036    2562 factory.go:153] Registering CRI-O factory
Sep 07 10:20:12.390059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390049    2562 factory.go:223] Registration of the crio container factory successfully
Sep 07 10:20:12.390378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390102    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 10:20:12.390378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390122    2562 factory.go:103] Registering Raw factory
Sep 07 10:20:12.390378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390136    2562 manager.go:1196] Started watching for new ooms in manager
Sep 07 10:20:12.390767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.390572    2562 manager.go:349] Starting recovery of all containers
Sep 07 10:20:12.397046 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.397023    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 10:20:12.397179 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.397025    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 10:20:12.398427 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.397060    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-36.ec2.internal.18d302ab727379c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-36.ec2.internal,UID:ip-10-0-142-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-36.ec2.internal,},FirstTimestamp:2026-09-07 10:20:12.379052489 +0000 UTC m=+0.389705794,LastTimestamp:2026-09-07 10:20:12.379052489 +0000 UTC m=+0.389705794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-36.ec2.internal,}"
Sep 07 10:20:12.401003 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.400988    2562 manager.go:354] Recovery completed
Sep 07 10:20:12.405479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.405435    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.407048 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407028    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.407118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407063    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.407118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407077    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.407590 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407577    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 10:20:12.407627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407591    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 10:20:12.407627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407581    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.407627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.407608    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:20:12.408356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.408336    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.408405 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.408367    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.408405 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.408377    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.409603 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.409591    2562 policy_none.go:49] "None policy: Start"
Sep 07 10:20:12.409638 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.409606    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 10:20:12.409638 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.409615    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 10:20:12.411593 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.411583    2562 policy_none.go:47] "Start"
Sep 07 10:20:12.450862 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.450845    2562 manager.go:337] "Starting Device Plugin manager"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.450887    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.450899    2562 server.go:87] "Starting device plugin registration server"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.451129    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.451141    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.451234    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.451337    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.451351    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.451835    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 10:20:12.470086 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.451881    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.488351 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.488331    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 10:20:12.489597 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.489577    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 10:20:12.489669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.489633    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 10:20:12.489669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.489656    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 10:20:12.489750 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.489697    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 10:20:12.492005 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.491990    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:20:12.551444 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.551391    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.552267 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.552252    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.552341 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.552281    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.552341 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.552295    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.552341 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.552339    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.560793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.560775    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.560848 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.560794    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-36.ec2.internal\": node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.572050 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.572035    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.595819 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.595787    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"]
Sep 07 10:20:12.595895 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.595862    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.597234 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.597220    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.597327 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.597248    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.597327 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.597261    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.599507 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.599493    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.599606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.599591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.599653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.599618    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.600158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600145    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.600215 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600170    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.600215 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600180    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.600740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600722    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.600814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600751    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.600814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.600763    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.602799 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.602786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.602901 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.602809    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:20:12.603493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.603477    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:20:12.603567 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.603503    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:20:12.603567 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.603512    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:20:12.628826 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.628810    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-36.ec2.internal\" not found" node="ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.632969 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.632954    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-36.ec2.internal\" not found" node="ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.672343 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.672321    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.772729 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.772704    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.790983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.790960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f124c2f31ba99ab09255e938134ba7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-36.ec2.internal\" (UID: \"e3f124c2f31ba99ab09255e938134ba7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.791040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.790995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.791040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.791012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.873399 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.873336    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:12.891669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.891713 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f124c2f31ba99ab09255e938134ba7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-36.ec2.internal\" (UID: \"e3f124c2f31ba99ab09255e938134ba7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.891713 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.891806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.891806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a765110ba6f08e1b9cbc3fb8e0804179-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal\" (UID: \"a765110ba6f08e1b9cbc3fb8e0804179\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.891806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.891774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e3f124c2f31ba99ab09255e938134ba7-config\") pod \"kube-apiserver-proxy-ip-10-0-142-36.ec2.internal\" (UID: \"e3f124c2f31ba99ab09255e938134ba7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.931808 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.931780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.935315 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:12.935280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:12.973812 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:12.973780    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.074257 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.074231    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.174845 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.174783    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.275381 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.275352    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.324864 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.324840    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 10:20:13.325253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.324977    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 10:20:13.376455 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.376319    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.397374 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.397332    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:15:12 +0000 UTC" deadline="2028-05-04 06:54:42.159873968 +0000 UTC"
Sep 07 10:20:13.397374 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.397371    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14516h34m28.762506991s"
Sep 07 10:20:13.476722 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.476653    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.497465 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:13.497434    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda765110ba6f08e1b9cbc3fb8e0804179.slice/crio-2aad4a3a0e21319b5540717950dc1b13cfc1fcbad9a9357d8bdf04d5bda573f5 WatchSource:0}: Error finding container 2aad4a3a0e21319b5540717950dc1b13cfc1fcbad9a9357d8bdf04d5bda573f5: Status 404 returned error can't find the container with id 2aad4a3a0e21319b5540717950dc1b13cfc1fcbad9a9357d8bdf04d5bda573f5
Sep 07 10:20:13.497958 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:13.497942    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f124c2f31ba99ab09255e938134ba7.slice/crio-fa5ae15f1a7c25a8d014d461968e9b43107b68f7e497df5004aa880ff85ad54c WatchSource:0}: Error finding container fa5ae15f1a7c25a8d014d461968e9b43107b68f7e497df5004aa880ff85ad54c: Status 404 returned error can't find the container with id fa5ae15f1a7c25a8d014d461968e9b43107b68f7e497df5004aa880ff85ad54c
Sep 07 10:20:13.501964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.501948    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:20:13.576956 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.576930    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.677343 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.677317    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.716782 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.716760    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:20:13.777851 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:13.777793    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-36.ec2.internal\" not found"
Sep 07 10:20:13.780454 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.780437    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:20:13.789771 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.789747    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:13.797494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.797477    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:20:13.798502 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.798487    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal"
Sep 07 10:20:13.807120 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.807106    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:20:13.811228 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:13.811215    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:20:14.367448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.367383    2562 apiserver.go:52] "Watching apiserver"
Sep 07 10:20:14.377730 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.377446    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 10:20:14.378215 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.378193    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9n2kt","openshift-network-operator/iptables-alerter-jqnbx","openshift-ovn-kubernetes/ovnkube-node-k52cb","kube-system/konnectivity-agent-84zcq","kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r","openshift-cluster-node-tuning-operator/tuned-zj6f6","openshift-dns/node-resolver-xl45g","openshift-image-registry/node-ca-cmtwk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal","openshift-multus/multus-6jvg7","kube-system/global-pull-secret-syncer-tc82s","openshift-multus/multus-additional-cni-plugins-tbw29","openshift-multus/network-metrics-daemon-zhvfh"]
Sep 07 10:20:14.380708 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.380678    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.382692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.382675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.384034 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.384019    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.385072 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.385049    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sb4v5\""
Sep 07 10:20:14.385157 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.385141    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.385212 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.385062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.388919 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.388889    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.389079 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.389344 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389327    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 10:20:14.389437 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389346    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fk67j\""
Sep 07 10:20:14.389437 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4gjq4\""
Sep 07 10:20:14.389767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 10:20:14.389767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.389767 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.389671    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:14.389767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389714    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 10:20:14.389959 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389882    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.389959 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.389934    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 10:20:14.390045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.390008    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 10:20:14.390284 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.390266    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.390562 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.390534    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 10:20:14.390625 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.390566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 10:20:14.390625 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.390583    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.391215 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.391187    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4zj8j\""
Sep 07 10:20:14.391311 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.391215    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 10:20:14.391587 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.391571    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 10:20:14.391750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.391730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.393961 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.393945    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.394138 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.394123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 10:20:14.394216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.394175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2dt68\""
Sep 07 10:20:14.394381 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.394366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.394451 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.394373    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.396203 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.396178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.396586 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.396567    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 10:20:14.396720 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.396707    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.396772 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.396731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.396885 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.396872    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhk84\""
Sep 07 10:20:14.398356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.398338    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 10:20:14.398565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.398543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d9b8l\""
Sep 07 10:20:14.398652 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.398631    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.398706 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.398684    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 10:20:14.398990 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.398973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-device-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.399062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-system-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.399062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-log-socket\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.399062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-socket-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.399200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399078    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.399200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsczz\" (UniqueName: \"kubernetes.io/projected/4c53941d-bca9-418f-b1a4-e4c346a08aa6-kube-api-access-hsczz\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.399200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.399200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-registration-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.399200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-sys\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.399500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-tuned\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.399500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-netns\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.399500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-slash\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.399500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ee1345-8c97-4dd4-8534-31c70c349b80-agent-certs\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.399500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399404    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-systemd\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-env-overrides\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5f7\" (UniqueName: \"kubernetes.io/projected/a17b86d7-2d73-4a68-b930-2c58ec10967a-kube-api-access-9m5f7\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-kubelet\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.399777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399719    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrxc\" (UniqueName: \"kubernetes.io/projected/3e92996d-bebd-40a6-9f37-771ea7981a16-kube-api-access-hjrxc\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-k8s-cni-cncf-io\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-ovn\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-conf\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-host\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399923    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-multus-certs\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-systemd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.399962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-node-log\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtw7q\" (UniqueName: \"kubernetes.io/projected/806e6219-5446-49d2-9873-097938288618-kube-api-access-gtw7q\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.400052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-modprobe-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysconfig\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovn-node-metrics-cert\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-sys-fs\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-kubernetes\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-bin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-kubelet\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-netns\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-bin\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-cni-binary-copy\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-conf-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ee1345-8c97-4dd4-8534-31c70c349b80-konnectivity-ca\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-run\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-lib-modules\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-hostroot\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.400559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400563    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-systemd-units\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-netd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400612    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-script-lib\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-var-lib-kubelet\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-cnibin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-socket-dir-parent\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-multus\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-config\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-tmp\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400869    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-daemon-config\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-etc-kubernetes\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.400928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-etc-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.401003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-var-lib-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.401016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7g2n\" (UniqueName: \"kubernetes.io/projected/1bdccc36-a482-400f-85d1-f9fcdbe5a277-kube-api-access-n7g2n\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.401773 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.401347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.402097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.402069    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.402097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.402092    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bj6mq\""
Sep 07 10:20:14.403063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.402281    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.403063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.402927    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 10:20:14.403580 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.403564    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:20:14.404360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.404337    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v7cms\""
Sep 07 10:20:14.404457 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.404408    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 10:20:14.405633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.405616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:14.406133 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.405620    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 10:20:14.406487 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.406468    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:14.409495 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.409478    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.409564 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.409530    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:14.422745 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.422728    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8n8nx"
Sep 07 10:20:14.427600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.427585    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8n8nx"
Sep 07 10:20:14.490919 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.490892    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 10:20:14.494212 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.494173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal" event={"ID":"a765110ba6f08e1b9cbc3fb8e0804179","Type":"ContainerStarted","Data":"2aad4a3a0e21319b5540717950dc1b13cfc1fcbad9a9357d8bdf04d5bda573f5"}
Sep 07 10:20:14.495200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.495174    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal" event={"ID":"e3f124c2f31ba99ab09255e938134ba7","Type":"ContainerStarted","Data":"fa5ae15f1a7c25a8d014d461968e9b43107b68f7e497df5004aa880ff85ad54c"}
Sep 07 10:20:14.501460 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-socket-dir-parent\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-multus\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501487    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-config\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-dbus\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-tmp\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.501559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-daemon-config\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-etc-kubernetes\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-etc-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nqf\" (UniqueName: \"kubernetes.io/projected/034465eb-ecc3-4b74-836b-772dfde3553b-kube-api-access-65nqf\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-var-lib-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7g2n\" (UniqueName: \"kubernetes.io/projected/1bdccc36-a482-400f-85d1-f9fcdbe5a277-kube-api-access-n7g2n\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/034465eb-ecc3-4b74-836b-772dfde3553b-host\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ee5f761-41ff-4ff9-96df-5356181ec1cc-host-slash\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777e3ff7-f089-4236-b771-d0d365587433-tmp-dir\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-device-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-system-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-log-socket\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-socket-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsczz\" (UniqueName: \"kubernetes.io/projected/4c53941d-bca9-418f-b1a4-e4c346a08aa6-kube-api-access-hsczz\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.501848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501834    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-registration-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-sys\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-tuned\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501913    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-netns\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-slash\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ee1345-8c97-4dd4-8534-31c70c349b80-agent-certs\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.501989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-systemd\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-env-overrides\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cnibin\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5f7\" (UniqueName: \"kubernetes.io/projected/a17b86d7-2d73-4a68-b930-2c58ec10967a-kube-api-access-9m5f7\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-kubelet\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/034465eb-ecc3-4b74-836b-772dfde3553b-serviceca\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrxc\" (UniqueName: \"kubernetes.io/projected/3e92996d-bebd-40a6-9f37-771ea7981a16-kube-api-access-hjrxc\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.502611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-k8s-cni-cncf-io\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-ovn\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-conf\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-host\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-multus-certs\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-systemd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-node-log\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-system-cni-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502373    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtw7q\" (UniqueName: \"kubernetes.io/projected/806e6219-5446-49d2-9873-097938288618-kube-api-access-gtw7q\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-modprobe-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysconfig\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovn-node-metrics-cert\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ee5f761-41ff-4ff9-96df-5356181ec1cc-iptables-alerter-script\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/777e3ff7-f089-4236-b771-d0d365587433-hosts-file\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.503365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-sys-fs\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-kubernetes\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-bin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-kubelet\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlgd\" (UniqueName: \"kubernetes.io/projected/777e3ff7-f089-4236-b771-d0d365587433-kube-api-access-5xlgd\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-netns\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-bin\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-kubelet-config\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-cni-binary-copy\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-conf-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ee1345-8c97-4dd4-8534-31c70c349b80-konnectivity-ca\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hp5\" (UniqueName: \"kubernetes.io/projected/1ee5f761-41ff-4ff9-96df-5356181ec1cc-kube-api-access-x9hp5\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-run\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-lib-modules\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-hostroot\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.503939 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-systemd-units\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-netd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-script-lib\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhmg\" (UniqueName: \"kubernetes.io/projected/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-kube-api-access-5vhmg\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-var-lib-kubelet\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.502995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-cnibin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-cnibin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-socket-dir-parent\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503239    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-multus\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-ovn\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-registration-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503674    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-sys\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-config\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.503980    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 10:20:14.504417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-conf\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-host\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-daemon-config\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-multus-certs\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-etc-kubernetes\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-etc-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-node-log\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-run-systemd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-var-lib-openvswitch\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-modprobe-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-netns\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-slash\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-device-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysconfig\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-system-cni-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504693    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-log-socket\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-socket-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-sysctl-d\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.505085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.504964    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-kubelet\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a17b86d7-2d73-4a68-b930-2c58ec10967a-cni-binary-copy\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505443    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-systemd\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-etc-selinux\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/806e6219-5446-49d2-9873-097938288618-sys-fs\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-kubernetes\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-cni-bin\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-var-lib-kubelet\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-host-run-k8s-cni-cncf-io\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-run-netns\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505741    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-bin\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-hostroot\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-run\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a17b86d7-2d73-4a68-b930-2c58ec10967a-multus-conf-dir\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-systemd-units\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bdccc36-a482-400f-85d1-f9fcdbe5a277-host-cni-netd\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.506565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.505999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-env-overrides\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.506427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/80ee1345-8c97-4dd4-8534-31c70c349b80-konnectivity-ca\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.506432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovnkube-script-lib\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.506498    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.506503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-var-lib-kubelet\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.506568    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:15.006550702 +0000 UTC m=+3.017204007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:14.507275 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.507018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c53941d-bca9-418f-b1a4-e4c346a08aa6-lib-modules\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.508640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.508619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-tmp\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.508801 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.508780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bdccc36-a482-400f-85d1-f9fcdbe5a277-ovn-node-metrics-cert\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.509750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.509720    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/80ee1345-8c97-4dd4-8534-31c70c349b80-agent-certs\") pod \"konnectivity-agent-84zcq\" (UID: \"80ee1345-8c97-4dd4-8534-31c70c349b80\") " pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.511176 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.511149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsczz\" (UniqueName: \"kubernetes.io/projected/4c53941d-bca9-418f-b1a4-e4c346a08aa6-kube-api-access-hsczz\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.511273 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.511259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c53941d-bca9-418f-b1a4-e4c346a08aa6-etc-tuned\") pod \"tuned-zj6f6\" (UID: \"4c53941d-bca9-418f-b1a4-e4c346a08aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.512100 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.512082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtw7q\" (UniqueName: \"kubernetes.io/projected/806e6219-5446-49d2-9873-097938288618-kube-api-access-gtw7q\") pod \"aws-ebs-csi-driver-node-zwz4r\" (UID: \"806e6219-5446-49d2-9873-097938288618\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.512354 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.512329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrxc\" (UniqueName: \"kubernetes.io/projected/3e92996d-bebd-40a6-9f37-771ea7981a16-kube-api-access-hjrxc\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:14.512944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.512925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7g2n\" (UniqueName: \"kubernetes.io/projected/1bdccc36-a482-400f-85d1-f9fcdbe5a277-kube-api-access-n7g2n\") pod \"ovnkube-node-k52cb\" (UID: \"1bdccc36-a482-400f-85d1-f9fcdbe5a277\") " pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.513914 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.513650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5f7\" (UniqueName: \"kubernetes.io/projected/a17b86d7-2d73-4a68-b930-2c58ec10967a-kube-api-access-9m5f7\") pod \"multus-6jvg7\" (UID: \"a17b86d7-2d73-4a68-b930-2c58ec10967a\") " pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.603756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ee5f761-41ff-4ff9-96df-5356181ec1cc-iptables-alerter-script\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.603756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/777e3ff7-f089-4236-b771-d0d365587433-hosts-file\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603824    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlgd\" (UniqueName: \"kubernetes.io/projected/777e3ff7-f089-4236-b771-d0d365587433-kube-api-access-5xlgd\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603837    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-kubelet-config\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hp5\" (UniqueName: \"kubernetes.io/projected/1ee5f761-41ff-4ff9-96df-5356181ec1cc-kube-api-access-x9hp5\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhmg\" (UniqueName: \"kubernetes.io/projected/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-kube-api-access-5vhmg\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-dbus\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.603971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nqf\" (UniqueName: \"kubernetes.io/projected/034465eb-ecc3-4b74-836b-772dfde3553b-kube-api-access-65nqf\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.603986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/034465eb-ecc3-4b74-836b-772dfde3553b-host\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ee5f761-41ff-4ff9-96df-5356181ec1cc-host-slash\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777e3ff7-f089-4236-b771-d0d365587433-tmp-dir\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cnibin\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/034465eb-ecc3-4b74-836b-772dfde3553b-serviceca\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-system-cni-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.604407 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-system-cni-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.604743 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.604727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1ee5f761-41ff-4ff9-96df-5356181ec1cc-iptables-alerter-script\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.605100 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.605079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-binary-copy\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.605156 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.605142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/777e3ff7-f089-4236-b771-d0d365587433-hosts-file\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.605553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.605530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-kubelet-config\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.605820 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.605795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.606024 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.606007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d824971e-4bb2-4633-b3b6-f705d282054c-dbus\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:14.606104 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.606084    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:14.606194 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.606134    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:15.10611965 +0000 UTC m=+3.116772954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:14.606731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.606711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.606813 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.606769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/034465eb-ecc3-4b74-836b-772dfde3553b-host\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.606873 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.606811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1ee5f761-41ff-4ff9-96df-5356181ec1cc-host-slash\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.607041 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.607026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777e3ff7-f089-4236-b771-d0d365587433-tmp-dir\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.607101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.607082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cnibin\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.607486 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.607466    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.607638 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.607624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/034465eb-ecc3-4b74-836b-772dfde3553b-serviceca\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.614190 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.614173    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:14.614271 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.614197    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:14.614271 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.614210    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:14.614271 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:14.614270    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:15.114256001 +0000 UTC m=+3.124909294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:14.616927 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.616887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hp5\" (UniqueName: \"kubernetes.io/projected/1ee5f761-41ff-4ff9-96df-5356181ec1cc-kube-api-access-x9hp5\") pod \"iptables-alerter-jqnbx\" (UID: \"1ee5f761-41ff-4ff9-96df-5356181ec1cc\") " pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.617186 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.617158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nqf\" (UniqueName: \"kubernetes.io/projected/034465eb-ecc3-4b74-836b-772dfde3553b-kube-api-access-65nqf\") pod \"node-ca-cmtwk\" (UID: \"034465eb-ecc3-4b74-836b-772dfde3553b\") " pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.617415 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.617394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlgd\" (UniqueName: \"kubernetes.io/projected/777e3ff7-f089-4236-b771-d0d365587433-kube-api-access-5xlgd\") pod \"node-resolver-xl45g\" (UID: \"777e3ff7-f089-4236-b771-d0d365587433\") " pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:14.618914 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.618866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhmg\" (UniqueName: \"kubernetes.io/projected/f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9-kube-api-access-5vhmg\") pod \"multus-additional-cni-plugins-tbw29\" (UID: \"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9\") " pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.693140 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.692943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zj6f6"
Sep 07 10:20:14.694239 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.694216    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:20:14.703392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.703366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6jvg7"
Sep 07 10:20:14.710787 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.710764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:14.715157 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.715142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:14.720553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.720531    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r"
Sep 07 10:20:14.726873 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.726855    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmtwk"
Sep 07 10:20:14.734328 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.734315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tbw29"
Sep 07 10:20:14.740930 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.740917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jqnbx"
Sep 07 10:20:14.746559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:14.746535    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xl45g"
Sep 07 10:20:15.106814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.106734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:15.106814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.106786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:15.107000 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.106943    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:15.107000 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.106944    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:15.107000 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.106998    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:16.106983734 +0000 UTC m=+4.117637026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:15.107117 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.107012    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:16.107006173 +0000 UTC m=+4.117659466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:15.207401 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.207356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:15.207565 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.207512    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:15.207565 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.207532    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:15.207565 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.207546    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:15.207696 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:15.207601    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:16.207587081 +0000 UTC m=+4.218240373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:15.339569 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.339541    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d31faf_f1f3_497a_a235_d1c7dd4fcdf9.slice/crio-a5369f6c67a1f2617156a087ca8f7e581dc582af374f7ff6cdfbaf91651716e7 WatchSource:0}: Error finding container a5369f6c67a1f2617156a087ca8f7e581dc582af374f7ff6cdfbaf91651716e7: Status 404 returned error can't find the container with id a5369f6c67a1f2617156a087ca8f7e581dc582af374f7ff6cdfbaf91651716e7
Sep 07 10:20:15.344713 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.344692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee5f761_41ff_4ff9_96df_5356181ec1cc.slice/crio-46eb20fdec2cfaa7544093327ae5f9b859d5b5dba5a4fd54d551462d0b3d7877 WatchSource:0}: Error finding container 46eb20fdec2cfaa7544093327ae5f9b859d5b5dba5a4fd54d551462d0b3d7877: Status 404 returned error can't find the container with id 46eb20fdec2cfaa7544093327ae5f9b859d5b5dba5a4fd54d551462d0b3d7877
Sep 07 10:20:15.346393 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.346274    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ee1345_8c97_4dd4_8534_31c70c349b80.slice/crio-65244f45a34d5d3adb9e321e886f4e71e37dfe2ad13aa20b56d60f7a5cbb9833 WatchSource:0}: Error finding container 65244f45a34d5d3adb9e321e886f4e71e37dfe2ad13aa20b56d60f7a5cbb9833: Status 404 returned error can't find the container with id 65244f45a34d5d3adb9e321e886f4e71e37dfe2ad13aa20b56d60f7a5cbb9833
Sep 07 10:20:15.348647 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.348336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806e6219_5446_49d2_9873_097938288618.slice/crio-dfcceb4f73f77444937a7b66445c6c030acc52409ce7e57ebbd2b4d74ecd2e5d WatchSource:0}: Error finding container dfcceb4f73f77444937a7b66445c6c030acc52409ce7e57ebbd2b4d74ecd2e5d: Status 404 returned error can't find the container with id dfcceb4f73f77444937a7b66445c6c030acc52409ce7e57ebbd2b4d74ecd2e5d
Sep 07 10:20:15.351595 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.351512    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777e3ff7_f089_4236_b771_d0d365587433.slice/crio-6e945469b8b9b97e528ebc6ac6d92affbb731acce1000d6e6362e5be0acbb08f WatchSource:0}: Error finding container 6e945469b8b9b97e528ebc6ac6d92affbb731acce1000d6e6362e5be0acbb08f: Status 404 returned error can't find the container with id 6e945469b8b9b97e528ebc6ac6d92affbb731acce1000d6e6362e5be0acbb08f
Sep 07 10:20:15.353232 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:15.353210    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034465eb_ecc3_4b74_836b_772dfde3553b.slice/crio-2786425c678be2c8a730a039d98ae6a46b85903b744be954e5a4afd5317ec36a WatchSource:0}: Error finding container 2786425c678be2c8a730a039d98ae6a46b85903b744be954e5a4afd5317ec36a: Status 404 returned error can't find the container with id 2786425c678be2c8a730a039d98ae6a46b85903b744be954e5a4afd5317ec36a
Sep 07 10:20:15.430883 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.430856    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:15:14 +0000 UTC" deadline="2028-05-18 11:31:28.371460758 +0000 UTC"
Sep 07 10:20:15.430883 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.430881    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14857h11m12.940582663s"
Sep 07 10:20:15.498789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.498759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal" event={"ID":"e3f124c2f31ba99ab09255e938134ba7","Type":"ContainerStarted","Data":"0fea62971a1aaa89669395f8337ec89ad3bf28b63d568e66488ccd953fd7b1d4"}
Sep 07 10:20:15.500045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.500025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"a0a9e626b9f7d0f959f51f4fd62e82d3d0b2b2a4a023d16913343feb75c31457"}
Sep 07 10:20:15.501752 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.501732    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmtwk" event={"ID":"034465eb-ecc3-4b74-836b-772dfde3553b","Type":"ContainerStarted","Data":"2786425c678be2c8a730a039d98ae6a46b85903b744be954e5a4afd5317ec36a"}
Sep 07 10:20:15.502852 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.502829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zj6f6" event={"ID":"4c53941d-bca9-418f-b1a4-e4c346a08aa6","Type":"ContainerStarted","Data":"50d495cd35db37c7f4e47a3d57faaf22476eb1418da9648e5d3d83f6b0ea4743"}
Sep 07 10:20:15.504027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.504008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6jvg7" event={"ID":"a17b86d7-2d73-4a68-b930-2c58ec10967a","Type":"ContainerStarted","Data":"c393123e5b58b7094593ad16b6d32d9cf687b9f602fbfad941ab8e9504eb2822"}
Sep 07 10:20:15.505037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.505011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r" event={"ID":"806e6219-5446-49d2-9873-097938288618","Type":"ContainerStarted","Data":"dfcceb4f73f77444937a7b66445c6c030acc52409ce7e57ebbd2b4d74ecd2e5d"}
Sep 07 10:20:15.506117 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.506092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerStarted","Data":"a5369f6c67a1f2617156a087ca8f7e581dc582af374f7ff6cdfbaf91651716e7"}
Sep 07 10:20:15.507122 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.507097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xl45g" event={"ID":"777e3ff7-f089-4236-b771-d0d365587433","Type":"ContainerStarted","Data":"6e945469b8b9b97e528ebc6ac6d92affbb731acce1000d6e6362e5be0acbb08f"}
Sep 07 10:20:15.508417 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.508398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-84zcq" event={"ID":"80ee1345-8c97-4dd4-8534-31c70c349b80","Type":"ContainerStarted","Data":"65244f45a34d5d3adb9e321e886f4e71e37dfe2ad13aa20b56d60f7a5cbb9833"}
Sep 07 10:20:15.509435 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:15.509415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqnbx" event={"ID":"1ee5f761-41ff-4ff9-96df-5356181ec1cc","Type":"ContainerStarted","Data":"46eb20fdec2cfaa7544093327ae5f9b859d5b5dba5a4fd54d551462d0b3d7877"}
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.130469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.130736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.130877    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.130936    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:18.130918104 +0000 UTC m=+6.141571398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.131392    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:16.131473 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.131450    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:18.131432217 +0000 UTC m=+6.142085515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:16.231437 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.231353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:16.231599 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.231554    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:16.231599 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.231573    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:16.231599 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.231584    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:16.231730 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.231639    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:18.231621723 +0000 UTC m=+6.242275034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:16.431356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.431279    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:15:14 +0000 UTC" deadline="2028-05-29 19:12:18.669293906 +0000 UTC"
Sep 07 10:20:16.431356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.431352    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15128h52m2.237946598s"
Sep 07 10:20:16.492917 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.492849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:16.493068 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.492973    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:16.493411 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.493389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:16.493546 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.493498    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:16.493604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.493570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:16.493686 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:16.493649    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:16.528543 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.527823    2562 generic.go:356] "Generic (PLEG): container finished" podID="a765110ba6f08e1b9cbc3fb8e0804179" containerID="1c3bf9f34f7af72c39f4eeb3f927e6dea4ccb0457585de9f1981a7a28d49925a" exitCode=0
Sep 07 10:20:16.528543 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.527964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal" event={"ID":"a765110ba6f08e1b9cbc3fb8e0804179","Type":"ContainerDied","Data":"1c3bf9f34f7af72c39f4eeb3f927e6dea4ccb0457585de9f1981a7a28d49925a"}
Sep 07 10:20:16.544165 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:16.544077    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-36.ec2.internal" podStartSLOduration=3.544050487 podStartE2EDuration="3.544050487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:15.519205071 +0000 UTC m=+3.529858385" watchObservedRunningTime="2026-09-07 10:20:16.544050487 +0000 UTC m=+4.554703801"
Sep 07 10:20:17.546615 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:17.546584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal" event={"ID":"a765110ba6f08e1b9cbc3fb8e0804179","Type":"ContainerStarted","Data":"7e37e1dbb4c686b53a227a04411b13fe43611cc3de2fd1e1480446a6f3d5166b"}
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.146540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.146591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.146745    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.146803    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:22.146786875 +0000 UTC m=+10.157440171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.146875    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:18.147162 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.146941    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:22.146922632 +0000 UTC m=+10.157575934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:18.247160 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.247121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:18.247360 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.247343    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:18.247416 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.247368    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:18.247416 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.247381    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:18.247519 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.247436    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:22.247419865 +0000 UTC m=+10.258073163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:18.490908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.490057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:18.490908 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.490248    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:18.490908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.490663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:18.490908 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.490754    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:18.490908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:18.490888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:18.491270 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:18.490997    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:20.490250 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:20.490219    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:20.490688 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:20.490363    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:20.490844 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:20.490791    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:20.490901 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:20.490882    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:20.491081 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:20.491063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:20.491233 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:20.491177    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.178526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.178578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.178712    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.178770    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:30.17875341 +0000 UTC m=+18.189406709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.179170    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:22.179250 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.179217    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:30.179202752 +0000 UTC m=+18.189856059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:22.279796 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.279760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:22.279986 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.279923    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:22.279986 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.279941    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:22.279986 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.279951    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:22.280136 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.280012    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:30.279995816 +0000 UTC m=+18.290649110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:22.490736 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.490660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:22.490888 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.490761    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:22.490888 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.490849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:22.490992 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.490974    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:22.491041 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:22.491008    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:22.491091 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:22.491078    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:24.491175 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:24.491139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:24.491633 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:24.491271    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:24.491633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:24.491356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:24.491804 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:24.491762    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:24.499290 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:24.499251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:24.499432 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:24.499373    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:26.490459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:26.490421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:26.490459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:26.490446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:26.490941 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:26.490525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:26.490941 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:26.490617    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:26.490941 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:26.490710    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:26.490941 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:26.490780    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:28.490748 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:28.490707    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:28.490748 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:28.490731    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:28.491234 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:28.490816    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:28.491234 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:28.490852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:28.491234 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:28.490957    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:28.491234 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:28.491038    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:30.240580 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.240534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:30.241096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.240590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:30.241096 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.240700    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:30.241096 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.240763    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:20:46.240743974 +0000 UTC m=+34.251397295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:30.241342 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.241130    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:30.241342 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.241171    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:46.241161279 +0000 UTC m=+34.251814571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:30.341912 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.341871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:30.342080 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.342010    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:30.342080 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.342025    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:30.342080 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.342035    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:30.342207 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.342089    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:20:46.34207162 +0000 UTC m=+34.352724926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.489906    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.489906    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.490027    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:30.490078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.490168    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:30.490352 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:30.490242    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:32.491472 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:32.491441    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:32.491861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:32.491544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:32.491861 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:32.491578    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:32.491861 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:32.491613    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:32.491861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:32.491636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:32.491861 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:32.491695    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:33.580945 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.580545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r" event={"ID":"806e6219-5446-49d2-9873-097938288618","Type":"ContainerStarted","Data":"e7ef025bb9e2417e1fa3302e945ca7df5cdfa445aa23868706c759b04c44f91c"}
Sep 07 10:20:33.582102 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.582074    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="88bec8204dd05b67119bc4ac34a978e4301214eb2c6b83ae8362b71a3d05974d" exitCode=0
Sep 07 10:20:33.582224 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.582169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"88bec8204dd05b67119bc4ac34a978e4301214eb2c6b83ae8362b71a3d05974d"}
Sep 07 10:20:33.584066 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.584044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xl45g" event={"ID":"777e3ff7-f089-4236-b771-d0d365587433","Type":"ContainerStarted","Data":"eaa19909b763cccd630223c6890edb7793e6a56120cdf65d8f6f640e9efc99b4"}
Sep 07 10:20:33.585882 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.585859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-84zcq" event={"ID":"80ee1345-8c97-4dd4-8534-31c70c349b80","Type":"ContainerStarted","Data":"f375a44ada710f6fcdab4dba822653bf09a250f53e3048fc2325c661b7723576"}
Sep 07 10:20:33.588905 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.588887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:20:33.589293 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589272    2562 generic.go:356] "Generic (PLEG): container finished" podID="1bdccc36-a482-400f-85d1-f9fcdbe5a277" containerID="899e19d8f771af5854ca4e4149bc69cc7c9b982c8f9ee910e65c221f74372cdd" exitCode=1
Sep 07 10:20:33.589398 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"cf13033528df63b20cd5ce503f5e46e13cfd81a1a78ec3cf9150709c6471afd9"}
Sep 07 10:20:33.589456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"7d97646fe4e953ca00659b0b813a378db0709b8f84fa6a246b5e365c64a06fa1"}
Sep 07 10:20:33.589509 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"5837217cc8c9c4343fe2b74c1ff4cbde86b5fa45bd434752fc7a2b68357eade9"}
Sep 07 10:20:33.589560 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"a223978f6e6d496054c12928fc8ff3882314ffd221bc3559491d8679af6f48fe"}
Sep 07 10:20:33.589611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerDied","Data":"899e19d8f771af5854ca4e4149bc69cc7c9b982c8f9ee910e65c221f74372cdd"}
Sep 07 10:20:33.589665 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.589620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"5ede30025aecf26e46217d77cb7a39c0224db19ceabd95e89692accac51cd7d9"}
Sep 07 10:20:33.593736 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.591263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmtwk" event={"ID":"034465eb-ecc3-4b74-836b-772dfde3553b","Type":"ContainerStarted","Data":"5cd440338f213f640682938ce2fb5eaf7d3d62e9b39c85a42e4502dee1229940"}
Sep 07 10:20:33.594732 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.594707    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zj6f6" event={"ID":"4c53941d-bca9-418f-b1a4-e4c346a08aa6","Type":"ContainerStarted","Data":"4f454ce1353a7cc5eb2fdc166034259c64b8ce50f8b53f7f95484b21c2718c32"}
Sep 07 10:20:33.596231 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.596211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6jvg7" event={"ID":"a17b86d7-2d73-4a68-b930-2c58ec10967a","Type":"ContainerStarted","Data":"1a7143038757e0b56a8b175a2f4530e658be2c2bf5396d142a5848b64d3e0ddf"}
Sep 07 10:20:33.605503 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.605469    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-36.ec2.internal" podStartSLOduration=20.605457201 podStartE2EDuration="20.605457201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:17.563499793 +0000 UTC m=+5.574153108" watchObservedRunningTime="2026-09-07 10:20:33.605457201 +0000 UTC m=+21.616110515"
Sep 07 10:20:33.618480 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.618438    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cmtwk" podStartSLOduration=4.271341353 podStartE2EDuration="21.618427489s" totalImagesPullingTime="17.347086136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:33.617986518 +0000 UTC m=+21.628639827" watchObservedRunningTime="2026-09-07 10:20:33.618427489 +0000 UTC m=+21.629080878"
Sep 07 10:20:33.637319 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.637261    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xl45g" podStartSLOduration=4.282718275 podStartE2EDuration="21.63724693s" totalImagesPullingTime="17.354528655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:33.636996399 +0000 UTC m=+21.647649715" watchObservedRunningTime="2026-09-07 10:20:33.63724693 +0000 UTC m=+21.647900248"
Sep 07 10:20:33.657210 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.657170    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zj6f6" podStartSLOduration=4.302712736 podStartE2EDuration="21.657159495s" totalImagesPullingTime="17.354446759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:33.656833063 +0000 UTC m=+21.667486377" watchObservedRunningTime="2026-09-07 10:20:33.657159495 +0000 UTC m=+21.667812808"
Sep 07 10:20:33.676281 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.676247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6jvg7" podStartSLOduration=3.77331271 podStartE2EDuration="21.676232338s" totalImagesPullingTime="17.902919628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:33.676036956 +0000 UTC m=+21.686690270" watchObservedRunningTime="2026-09-07 10:20:33.676232338 +0000 UTC m=+21.686885653"
Sep 07 10:20:33.690593 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:33.690555    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-84zcq" podStartSLOduration=8.018101649 podStartE2EDuration="21.690540355s" totalImagesPullingTime="13.672438706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:33.690168597 +0000 UTC m=+21.700821912" watchObservedRunningTime="2026-09-07 10:20:33.690540355 +0000 UTC m=+21.701193670"
Sep 07 10:20:34.133998 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.133977    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 10:20:34.463961 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.463870    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T10:20:34.13399417Z","UUID":"222f38f2-3036-4915-8669-165b66a524e1","Handler":null,"Name":"","Endpoint":""}
Sep 07 10:20:34.465635 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.465602    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 10:20:34.465635 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.465637    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 10:20:34.490795 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.490767    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:34.490795 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.490784    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:34.490953 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:34.490898    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:34.491023 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:34.490991    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:34.491079 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.491070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:34.491198 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:34.491175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:34.600457 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.600429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r" event={"ID":"806e6219-5446-49d2-9873-097938288618","Type":"ContainerStarted","Data":"5977db945c56f4b204035a905af122c5f38a2f9ef532b3c30046c0c3315f1b38"}
Sep 07 10:20:34.602360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.602337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jqnbx" event={"ID":"1ee5f761-41ff-4ff9-96df-5356181ec1cc","Type":"ContainerStarted","Data":"a336c3fbbb4cec2805c918893805474376a4d6f92bd60add6292f605f9a58d32"}
Sep 07 10:20:34.619555 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:34.619506    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jqnbx" podStartSLOduration=5.26129511 podStartE2EDuration="22.61948766s" totalImagesPullingTime="17.35819255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:34.619011486 +0000 UTC m=+22.629664801" watchObservedRunningTime="2026-09-07 10:20:34.61948766 +0000 UTC m=+22.630140975"
Sep 07 10:20:35.608995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:35.608934    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:20:35.609470 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:35.609416    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"9418f69b95310fdf173f29155a8ee1109630d9e268f764029570cb02768dc058"}
Sep 07 10:20:35.611734 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:35.611711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r" event={"ID":"806e6219-5446-49d2-9873-097938288618","Type":"ContainerStarted","Data":"31ff0ed281672558081951fa6ae8cbff739b89b4fcc8f679c31d692a00163ce6"}
Sep 07 10:20:35.630148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:35.630108    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zwz4r" podStartSLOduration=3.9688588339999997 podStartE2EDuration="23.630091687s" totalImagesPullingTime="19.661232853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:35.62992205 +0000 UTC m=+23.640575364" watchObservedRunningTime="2026-09-07 10:20:35.630091687 +0000 UTC m=+23.640745000"
Sep 07 10:20:36.490569 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:36.490541    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:36.490728 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:36.490547    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:36.490728 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:36.490637    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:36.490728 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:36.490695    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:36.490728 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:36.490554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:36.490873 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:36.490787    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:38.460134 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.459971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:38.460880 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.460489    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:38.490474 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.490448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:38.490474 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.490472    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:38.490608 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.490450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:38.490608 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:38.490545    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:38.490696 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:38.490615    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:38.490696 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:38.490676    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:38.619986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.619935    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="dc1607af01bb23b433b5af46e69e28509dbf13732e12bf185352117a1f7d08e3" exitCode=0
Sep 07 10:20:38.620076 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.620023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"dc1607af01bb23b433b5af46e69e28509dbf13732e12bf185352117a1f7d08e3"}
Sep 07 10:20:38.623630 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.623614    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:20:38.624026 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.624003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"69a7bf385f22948182d875ef36616953d52513c35238dc9d2ddd89d5deaa7e1c"}
Sep 07 10:20:38.624458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:38.624445    2562 scope.go:117] "RemoveContainer" containerID="899e19d8f771af5854ca4e4149bc69cc7c9b982c8f9ee910e65c221f74372cdd"
Sep 07 10:20:39.631742 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.631667    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:20:39.632336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.632311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" event={"ID":"1bdccc36-a482-400f-85d1-f9fcdbe5a277","Type":"ContainerStarted","Data":"27580455519f2b80b0b5abd2417cab01ff6cbabba23c4e58926bc144e1eb1214"}
Sep 07 10:20:39.632676 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.632653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:39.632787 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.632755    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:39.632843 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.632806    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:39.652728 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.652701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:39.652831 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.652804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:20:39.662444 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.662398    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb" podStartSLOduration=10.273447765 podStartE2EDuration="27.66238648s" totalImagesPullingTime="17.388938715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:39.658873091 +0000 UTC m=+27.669526406" watchObservedRunningTime="2026-09-07 10:20:39.66238648 +0000 UTC m=+27.673039815"
Sep 07 10:20:39.939644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.939573    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9n2kt"]
Sep 07 10:20:39.939818 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.939719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:39.939879 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:39.939813    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:39.942812 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.942787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zhvfh"]
Sep 07 10:20:39.942959 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.942893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:39.943021 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:39.942970    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:39.953148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.953127    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tc82s"]
Sep 07 10:20:39.953266 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:39.953215    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:39.953331 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:39.953288    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:40.636437 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:40.636410    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="63e730adaf04fbc83e000bc8442334da62cf264a2f3036dcd79b17a1b4fd1d7a" exitCode=0
Sep 07 10:20:40.636886 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:40.636493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"63e730adaf04fbc83e000bc8442334da62cf264a2f3036dcd79b17a1b4fd1d7a"}
Sep 07 10:20:41.490654 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:41.490628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:41.490789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:41.490629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:41.490789 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:41.490730    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:41.490875 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:41.490629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:41.490875 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:41.490795    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:41.490875 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:41.490855    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:42.642727 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:42.642561    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="592f1e87a6d3577e78196eb9d1a94c0112e1cfc01a4f6f52e22f13b76aa9854f" exitCode=0
Sep 07 10:20:42.642727 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:42.642645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"592f1e87a6d3577e78196eb9d1a94c0112e1cfc01a4f6f52e22f13b76aa9854f"}
Sep 07 10:20:42.695238 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:42.695214    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:42.695372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:42.695327    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 07 10:20:42.695803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:42.695783    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-84zcq"
Sep 07 10:20:43.489870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:43.489841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:43.490095 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:43.489884    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:43.490095 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:43.489969    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:43.490275 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:43.490260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:43.490365 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:43.490349    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:43.490487 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:43.490461    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:45.305444 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:45.305413    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xl45g_777e3ff7-f089-4236-b771-d0d365587433/dns-node-resolver/0.log"
Sep 07 10:20:45.490375 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:45.490348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:45.490531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:45.490356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:45.490531 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:45.490444    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tc82s" podUID="d824971e-4bb2-4633-b3b6-f705d282054c"
Sep 07 10:20:45.490634 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:45.490540    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9n2kt" podUID="dee58893-1ba9-408d-8e4d-1c5853c16a5d"
Sep 07 10:20:45.490634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:45.490356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:45.490703 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:45.490647    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zhvfh" podUID="3e92996d-bebd-40a6-9f37-771ea7981a16"
Sep 07 10:20:45.873769 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:45.873739    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmtwk_034465eb-ecc3-4b74-836b-772dfde3553b/node-ca/0.log"
Sep 07 10:20:46.229946 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.229918    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-36.ec2.internal" event="NodeReady"
Sep 07 10:20:46.230109 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.230038    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 10:20:46.260699 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.260670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:46.260863 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.260710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:46.260863 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.260820    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:46.260863 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.260821    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:46.261005 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.260867    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret podName:d824971e-4bb2-4633-b3b6-f705d282054c nodeName:}" failed. No retries permitted until 2026-09-07 10:21:18.260854573 +0000 UTC m=+66.271507865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret") pod "global-pull-secret-syncer-tc82s" (UID: "d824971e-4bb2-4633-b3b6-f705d282054c") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:20:46.261005 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.260882    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs podName:3e92996d-bebd-40a6-9f37-771ea7981a16 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:18.260874047 +0000 UTC m=+66.271527339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs") pod "network-metrics-daemon-zhvfh" (UID: "3e92996d-bebd-40a6-9f37-771ea7981a16") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:20:46.265965 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.265940    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:20:46.270983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.270965    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.273413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.273391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 10:20:46.273607 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.273588    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 10:20:46.273718 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.273680    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 10:20:46.273986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.273968    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z7qs2\""
Sep 07 10:20:46.280243 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.280074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 10:20:46.280477 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.280458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:20:46.284245 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.284224    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sd5vw"]
Sep 07 10:20:46.287785 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.287766    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hvdjj"]
Sep 07 10:20:46.287925 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.287905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.290641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 10:20:46.290641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290474    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 10:20:46.290641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290531    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 10:20:46.290641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290544    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xz5mc\""
Sep 07 10:20:46.290892 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290790    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 10:20:46.290971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.290955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.293618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.293409    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 10:20:46.293618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.293475    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 10:20:46.293618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.293554    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-594j9\""
Sep 07 10:20:46.293784 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.293684    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 10:20:46.297080 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.297059    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hvdjj"]
Sep 07 10:20:46.298830 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.298814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sd5vw"]
Sep 07 10:20:46.304797 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.304776    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6m9dk"]
Sep 07 10:20:46.314105 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.314085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.316830 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.316808    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 10:20:46.316929 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.316820    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkvfh\""
Sep 07 10:20:46.317112 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.317095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 10:20:46.317416 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.317395    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6m9dk"]
Sep 07 10:20:46.361944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.361918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.361953    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.361988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:46.362086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkglt\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362086 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.362078    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.362095    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.362104    2562 projected.go:196] Error preparing data for projected volume kube-api-access-jfqf2 for pod openshift-network-diagnostics/network-check-target-9n2kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362143    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.362215    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2 podName:dee58893-1ba9-408d-8e4d-1c5853c16a5d nodeName:}" failed. No retries permitted until 2026-09-07 10:21:18.36218798 +0000 UTC m=+66.372841288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jfqf2" (UniqueName: "kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2") pod "network-check-target-9n2kt" (UID: "dee58893-1ba9-408d-8e4d-1c5853c16a5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:20:46.362272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.362260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.382575 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.382552    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-qzfxc"]
Sep 07 10:20:46.385806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.385781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.388397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.388148    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 07 10:20:46.388397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.388210    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 07 10:20:46.388397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.388232    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mztf4\""
Sep 07 10:20:46.388397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.388291    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 07 10:20:46.388731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.388712    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 07 10:20:46.394002 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.393983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-qzfxc"]
Sep 07 10:20:46.463115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkglt\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9991cffd-bfa1-4c64-8e07-4800532d94e4-data-volume\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9991cffd-bfa1-4c64-8e07-4800532d94e4-crio-socket\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8fg8\" (UniqueName: \"kubernetes.io/projected/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-api-access-c8fg8\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.463353 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.463351    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.463370    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b47bc4647-ghs48: secret "image-registry-tls" not found
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.463436    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls podName:4a1827ec-75e6-47bc-baf6-3e102837195f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:46.96341585 +0000 UTC m=+34.974069161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls") pod "image-registry-7b47bc4647-ghs48" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f") : secret "image-registry-tls" not found
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/245a647f-a221-48f1-8175-37626932d896-config-volume\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fjx\" (UniqueName: \"kubernetes.io/projected/245a647f-a221-48f1-8175-37626932d896-kube-api-access-42fjx\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22vnw\" (UniqueName: \"kubernetes.io/projected/5847744b-f489-40d7-a83e-04fa5553b601-kube-api-access-22vnw\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.463653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.464096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5847744b-f489-40d7-a83e-04fa5553b601-cert\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.464096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.464096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245a647f-a221-48f1-8175-37626932d896-tmp-dir\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.464096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.463950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.464287 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.464152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.464608 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.464588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.464748 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.464640    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.468289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.468268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.468374 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.468274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.471427 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.471409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.471512 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.471476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkglt\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.564857 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.564793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5847744b-f489-40d7-a83e-04fa5553b601-cert\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.564857 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.564825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245a647f-a221-48f1-8175-37626932d896-tmp-dir\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.564857 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.564847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565072 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.564944    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:46.565072 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.565005    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:47.064984733 +0000 UTC m=+35.075638025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/245a647f-a221-48f1-8175-37626932d896-tmp-dir\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9991cffd-bfa1-4c64-8e07-4800532d94e4-data-volume\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffe6b463-8342-47c4-b217-b23fc3266866-signing-key\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9991cffd-bfa1-4c64-8e07-4800532d94e4-crio-socket\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8fg8\" (UniqueName: \"kubernetes.io/projected/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-api-access-c8fg8\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbgh\" (UniqueName: \"kubernetes.io/projected/ffe6b463-8342-47c4-b217-b23fc3266866-kube-api-access-tpbgh\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffe6b463-8342-47c4-b217-b23fc3266866-signing-cabundle\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/245a647f-a221-48f1-8175-37626932d896-config-volume\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fjx\" (UniqueName: \"kubernetes.io/projected/245a647f-a221-48f1-8175-37626932d896-kube-api-access-42fjx\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565657    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9991cffd-bfa1-4c64-8e07-4800532d94e4-crio-socket\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22vnw\" (UniqueName: \"kubernetes.io/projected/5847744b-f489-40d7-a83e-04fa5553b601-kube-api-access-22vnw\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.565770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9991cffd-bfa1-4c64-8e07-4800532d94e4-data-volume\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.565920    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.565921    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:20:46.565992 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.565969    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls podName:245a647f-a221-48f1-8175-37626932d896 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:47.065954159 +0000 UTC m=+35.076607452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls") pod "dns-default-6m9dk" (UID: "245a647f-a221-48f1-8175-37626932d896") : secret "dns-default-metrics-tls" not found
Sep 07 10:20:46.566720 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.566045    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:47.066020664 +0000 UTC m=+35.076673961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : configmap "kube-rbac-proxy" not found
Sep 07 10:20:46.566720 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.566181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/245a647f-a221-48f1-8175-37626932d896-config-volume\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.567546 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.567526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5847744b-f489-40d7-a83e-04fa5553b601-cert\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.573844 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.573819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8fg8\" (UniqueName: \"kubernetes.io/projected/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-api-access-c8fg8\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:46.574785 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.574765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fjx\" (UniqueName: \"kubernetes.io/projected/245a647f-a221-48f1-8175-37626932d896-kube-api-access-42fjx\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:46.575435 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.575415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22vnw\" (UniqueName: \"kubernetes.io/projected/5847744b-f489-40d7-a83e-04fa5553b601-kube-api-access-22vnw\") pod \"ingress-canary-hvdjj\" (UID: \"5847744b-f489-40d7-a83e-04fa5553b601\") " pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.609764 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.609740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hvdjj"
Sep 07 10:20:46.666518 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.666492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbgh\" (UniqueName: \"kubernetes.io/projected/ffe6b463-8342-47c4-b217-b23fc3266866-kube-api-access-tpbgh\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.666655 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.666526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffe6b463-8342-47c4-b217-b23fc3266866-signing-cabundle\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.666655 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.666592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffe6b463-8342-47c4-b217-b23fc3266866-signing-key\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.667348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.667321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ffe6b463-8342-47c4-b217-b23fc3266866-signing-cabundle\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.669133 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.669111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ffe6b463-8342-47c4-b217-b23fc3266866-signing-key\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.675452 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.675426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbgh\" (UniqueName: \"kubernetes.io/projected/ffe6b463-8342-47c4-b217-b23fc3266866-kube-api-access-tpbgh\") pod \"service-ca-75d99c484-qzfxc\" (UID: \"ffe6b463-8342-47c4-b217-b23fc3266866\") " pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.695370 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.695336    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-qzfxc"
Sep 07 10:20:46.972100 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:46.970888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:46.972100 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.971080    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:20:46.972100 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.971094    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b47bc4647-ghs48: secret "image-registry-tls" not found
Sep 07 10:20:46.972100 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:46.971166    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls podName:4a1827ec-75e6-47bc-baf6-3e102837195f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:47.971148445 +0000 UTC m=+35.981801758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls") pod "image-registry-7b47bc4647-ghs48" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f") : secret "image-registry-tls" not found
Sep 07 10:20:47.071746 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.071568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:47.071848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.071765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:47.071848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.071801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:47.071848 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071700    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:20:47.071997 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071867    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls podName:245a647f-a221-48f1-8175-37626932d896 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:48.07184724 +0000 UTC m=+36.082500551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls") pod "dns-default-6m9dk" (UID: "245a647f-a221-48f1-8175-37626932d896") : secret "dns-default-metrics-tls" not found
Sep 07 10:20:47.071997 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071875    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:20:47.071997 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071907    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:47.071997 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071924    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:48.071908449 +0000 UTC m=+36.082561759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : configmap "kube-rbac-proxy" not found
Sep 07 10:20:47.071997 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.071943    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:48.071932229 +0000 UTC m=+36.082585525 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:47.195603 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.195512    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-qzfxc"]
Sep 07 10:20:47.198989 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.198963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hvdjj"]
Sep 07 10:20:47.202587 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:47.202563    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe6b463_8342_47c4_b217_b23fc3266866.slice/crio-e9a2faea611dfce25e8cbf2bcd4170d1f54f039438d0e7e318ee37abbd1613b4 WatchSource:0}: Error finding container e9a2faea611dfce25e8cbf2bcd4170d1f54f039438d0e7e318ee37abbd1613b4: Status 404 returned error can't find the container with id e9a2faea611dfce25e8cbf2bcd4170d1f54f039438d0e7e318ee37abbd1613b4
Sep 07 10:20:47.203381 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:20:47.203358    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5847744b_f489_40d7_a83e_04fa5553b601.slice/crio-5def4e5c7ec8a0d05ea0b99890472bf21df0aa389b484194458e59f8f2121f33 WatchSource:0}: Error finding container 5def4e5c7ec8a0d05ea0b99890472bf21df0aa389b484194458e59f8f2121f33: Status 404 returned error can't find the container with id 5def4e5c7ec8a0d05ea0b99890472bf21df0aa389b484194458e59f8f2121f33
Sep 07 10:20:47.490791 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.490726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:20:47.491376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.490812    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:20:47.491376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.490837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:20:47.494968 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.494947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:20:47.495115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.494972    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jrgm\""
Sep 07 10:20:47.495115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.495000    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:20:47.495115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.495001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbx74\""
Sep 07 10:20:47.495115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.495041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:20:47.495115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.494951    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:20:47.655350 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.655321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-qzfxc" event={"ID":"ffe6b463-8342-47c4-b217-b23fc3266866","Type":"ContainerStarted","Data":"e9a2faea611dfce25e8cbf2bcd4170d1f54f039438d0e7e318ee37abbd1613b4"}
Sep 07 10:20:47.656439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.656412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hvdjj" event={"ID":"5847744b-f489-40d7-a83e-04fa5553b601","Type":"ContainerStarted","Data":"5def4e5c7ec8a0d05ea0b99890472bf21df0aa389b484194458e59f8f2121f33"}
Sep 07 10:20:47.658982 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.658961    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="7b2089216800e81df2b03b22931b6e20a6de7fa953a5bbc75eac23a3b1a876f3" exitCode=0
Sep 07 10:20:47.659090 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.658987    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"7b2089216800e81df2b03b22931b6e20a6de7fa953a5bbc75eac23a3b1a876f3"}
Sep 07 10:20:47.980739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:47.980701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:47.980908 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.980869    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:20:47.980908 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.980885    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b47bc4647-ghs48: secret "image-registry-tls" not found
Sep 07 10:20:47.981013 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:47.980942    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls podName:4a1827ec-75e6-47bc-baf6-3e102837195f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:49.980924016 +0000 UTC m=+37.991577308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls") pod "image-registry-7b47bc4647-ghs48" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f") : secret "image-registry-tls" not found
Sep 07 10:20:48.081937 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.081858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:48.082090 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.081978    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:48.082090 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.082005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:48.082090 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:48.082010    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:48.082090 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:48.082072    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:50.082054137 +0000 UTC m=+38.092707449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:48.082230 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:48.082125    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:20:48.082230 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:48.082187    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls podName:245a647f-a221-48f1-8175-37626932d896 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:50.082171941 +0000 UTC m=+38.092825235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls") pod "dns-default-6m9dk" (UID: "245a647f-a221-48f1-8175-37626932d896") : secret "dns-default-metrics-tls" not found
Sep 07 10:20:48.082573 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.082551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9991cffd-bfa1-4c64-8e07-4800532d94e4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:48.666034 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.665985    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9" containerID="726f08ee3f1206e1c74539e156481a013a4434e69585122a1f9ff98ed386da48" exitCode=0
Sep 07 10:20:48.666413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:48.666051    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerDied","Data":"726f08ee3f1206e1c74539e156481a013a4434e69585122a1f9ff98ed386da48"}
Sep 07 10:20:49.672070 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:49.672039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tbw29" event={"ID":"f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9","Type":"ContainerStarted","Data":"5ff8197f6507db46e8dc472dc29c8963d18541150c700cdebc74a64d27275078"}
Sep 07 10:20:49.693191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:49.693129    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tbw29" podStartSLOduration=9.272353589 podStartE2EDuration="37.693111573s" totalImagesPullingTime="28.420757984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:49.69112637 +0000 UTC m=+37.701779684" watchObservedRunningTime="2026-09-07 10:20:49.693111573 +0000 UTC m=+37.703764881"
Sep 07 10:20:50.001731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.001688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:50.001891 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.001818    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:20:50.001891 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.001837    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b47bc4647-ghs48: secret "image-registry-tls" not found
Sep 07 10:20:50.001891 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.001890    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls podName:4a1827ec-75e6-47bc-baf6-3e102837195f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:54.00187627 +0000 UTC m=+42.012529563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls") pod "image-registry-7b47bc4647-ghs48" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f") : secret "image-registry-tls" not found
Sep 07 10:20:50.102983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.102952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:50.103116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.102993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:50.103116 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.103091    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:20:50.103116 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.103101    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:50.103205 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.103147    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:54.103135761 +0000 UTC m=+42.113789054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:50.103205 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:50.103158    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls podName:245a647f-a221-48f1-8175-37626932d896 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:54.103152833 +0000 UTC m=+42.113806125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls") pod "dns-default-6m9dk" (UID: "245a647f-a221-48f1-8175-37626932d896") : secret "dns-default-metrics-tls" not found
Sep 07 10:20:50.677600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.677129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-qzfxc" event={"ID":"ffe6b463-8342-47c4-b217-b23fc3266866","Type":"ContainerStarted","Data":"ae15bd06c8399ac6ae3b2ca4cf535864f98444bb00aab8ecd30983e95ce07328"}
Sep 07 10:20:50.679126 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.679099    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hvdjj" event={"ID":"5847744b-f489-40d7-a83e-04fa5553b601","Type":"ContainerStarted","Data":"f170b8d756883e0a1820477605de7c2b63748f49c0a7f691d17e057c2608f942"}
Sep 07 10:20:50.694137 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.694079    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-qzfxc" podStartSLOduration=1.784833573 podStartE2EDuration="4.694059835s" totalImagesPullingTime="2.909226262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:50.692781977 +0000 UTC m=+38.703435288" watchObservedRunningTime="2026-09-07 10:20:50.694059835 +0000 UTC m=+38.704713150"
Sep 07 10:20:50.712610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:50.712559    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hvdjj" podStartSLOduration=1.4528392399999999 podStartE2EDuration="4.712538676s" totalImagesPullingTime="3.259699436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:50.712417476 +0000 UTC m=+38.723070791" watchObservedRunningTime="2026-09-07 10:20:50.712538676 +0000 UTC m=+38.723191991"
Sep 07 10:20:54.034250 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:54.034216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:20:54.034660 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.034382    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:20:54.034660 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.034400    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b47bc4647-ghs48: secret "image-registry-tls" not found
Sep 07 10:20:54.034660 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.034452    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls podName:4a1827ec-75e6-47bc-baf6-3e102837195f nodeName:}" failed. No retries permitted until 2026-09-07 10:21:02.034437818 +0000 UTC m=+50.045091111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls") pod "image-registry-7b47bc4647-ghs48" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f") : secret "image-registry-tls" not found
Sep 07 10:20:54.134793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:54.134762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:20:54.134925 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:20:54.134801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:20:54.134925 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.134900    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:54.134925 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.134911    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:20:54.135029 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.134950    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls podName:9991cffd-bfa1-4c64-8e07-4800532d94e4 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:02.134936864 +0000 UTC m=+50.145590156 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sd5vw" (UID: "9991cffd-bfa1-4c64-8e07-4800532d94e4") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:20:54.135029 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:20:54.134973    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls podName:245a647f-a221-48f1-8175-37626932d896 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:02.134956877 +0000 UTC m=+50.145610176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls") pod "dns-default-6m9dk" (UID: "245a647f-a221-48f1-8175-37626932d896") : secret "dns-default-metrics-tls" not found
Sep 07 10:21:02.089531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.089496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:02.093497 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.093470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"image-registry-7b47bc4647-ghs48\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") " pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:02.182288 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.182261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:02.190533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.190510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:21:02.190641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.190545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:21:02.192995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.192840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9991cffd-bfa1-4c64-8e07-4800532d94e4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sd5vw\" (UID: \"9991cffd-bfa1-4c64-8e07-4800532d94e4\") " pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:21:02.192995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.192840    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/245a647f-a221-48f1-8175-37626932d896-metrics-tls\") pod \"dns-default-6m9dk\" (UID: \"245a647f-a221-48f1-8175-37626932d896\") " pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:21:02.200822 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.200793    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sd5vw"
Sep 07 10:21:02.224083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.224060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:21:02.473124 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.473054    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:21:02.476073 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:02.476050    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1827ec_75e6_47bc_baf6_3e102837195f.slice/crio-e1f572cac3cb28414c865ec88ca22c013b8081208994bc2bbabb1cffe1a87d8d WatchSource:0}: Error finding container e1f572cac3cb28414c865ec88ca22c013b8081208994bc2bbabb1cffe1a87d8d: Status 404 returned error can't find the container with id e1f572cac3cb28414c865ec88ca22c013b8081208994bc2bbabb1cffe1a87d8d
Sep 07 10:21:02.497295 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.497273    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sd5vw"]
Sep 07 10:21:02.499434 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:02.499407    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9991cffd_bfa1_4c64_8e07_4800532d94e4.slice/crio-1ab70fbbf0318a2d88c98fefab72247fa057033cd766f60bd64afd35ba3db924 WatchSource:0}: Error finding container 1ab70fbbf0318a2d88c98fefab72247fa057033cd766f60bd64afd35ba3db924: Status 404 returned error can't find the container with id 1ab70fbbf0318a2d88c98fefab72247fa057033cd766f60bd64afd35ba3db924
Sep 07 10:21:02.520087 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.520065    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6m9dk"]
Sep 07 10:21:02.523344 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:02.523318    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245a647f_a221_48f1_8175_37626932d896.slice/crio-c9002fd422ebaf3d8b1424e2b09b78802de69e96db53cfd02b7d7a02f93f9fc5 WatchSource:0}: Error finding container c9002fd422ebaf3d8b1424e2b09b78802de69e96db53cfd02b7d7a02f93f9fc5: Status 404 returned error can't find the container with id c9002fd422ebaf3d8b1424e2b09b78802de69e96db53cfd02b7d7a02f93f9fc5
Sep 07 10:21:02.707463 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.707420    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6m9dk" event={"ID":"245a647f-a221-48f1-8175-37626932d896","Type":"ContainerStarted","Data":"c9002fd422ebaf3d8b1424e2b09b78802de69e96db53cfd02b7d7a02f93f9fc5"}
Sep 07 10:21:02.709016 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.708990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sd5vw" event={"ID":"9991cffd-bfa1-4c64-8e07-4800532d94e4","Type":"ContainerStarted","Data":"68debd9fcab4b0ca700e6e69e19ec9d2f5d927f2dfb8d4c2b9e82be06fb8f0eb"}
Sep 07 10:21:02.709135 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.709048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sd5vw" event={"ID":"9991cffd-bfa1-4c64-8e07-4800532d94e4","Type":"ContainerStarted","Data":"1ab70fbbf0318a2d88c98fefab72247fa057033cd766f60bd64afd35ba3db924"}
Sep 07 10:21:02.710529 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.710507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" event={"ID":"4a1827ec-75e6-47bc-baf6-3e102837195f","Type":"ContainerStarted","Data":"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"}
Sep 07 10:21:02.710609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.710545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" event={"ID":"4a1827ec-75e6-47bc-baf6-3e102837195f","Type":"ContainerStarted","Data":"e1f572cac3cb28414c865ec88ca22c013b8081208994bc2bbabb1cffe1a87d8d"}
Sep 07 10:21:02.710753 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.710737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:02.728668 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:02.728592    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" podStartSLOduration=51.728574345 podStartE2EDuration="51.728574345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:02.727961307 +0000 UTC m=+50.738614635" watchObservedRunningTime="2026-09-07 10:21:02.728574345 +0000 UTC m=+50.739227654"
Sep 07 10:21:04.015277 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.015202    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"]
Sep 07 10:21:04.039001 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.038980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"]
Sep 07 10:21:04.039127 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.039097    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.042243 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.042224    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 10:21:04.042388 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.042226    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 10:21:04.042614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.042600    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 10:21:04.042857 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.042842    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9rzks\""
Sep 07 10:21:04.044946 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.044927    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 07 10:21:04.054996 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.054971    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"]
Sep 07 10:21:04.055086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.054978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.061366 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.058598    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 07 10:21:04.077012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.076988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"]
Sep 07 10:21:04.077012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.077015    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"]
Sep 07 10:21:04.077174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.077033    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"]
Sep 07 10:21:04.077174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.077047    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:21:04.077174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.077125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.079509 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.079489    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 10:21:04.080143 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.080123    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 10:21:04.080320 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.080285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 10:21:04.080393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.080345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 10:21:04.105003 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.104978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/763acf38-09b1-446d-b660-edbae66af565-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.105125 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gww5c\" (UniqueName: \"kubernetes.io/projected/763acf38-09b1-446d-b660-edbae66af565-kube-api-access-gww5c\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.105125 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.105125 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.105125 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105094    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105131    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8059065a-ceda-4990-a33c-476e8c5d4300-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105174    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763acf38-09b1-446d-b660-edbae66af565-tmp\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwn2\" (UniqueName: \"kubernetes.io/projected/8059065a-ceda-4990-a33c-476e8c5d4300-kube-api-access-njwn2\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twpf\" (UniqueName: \"kubernetes.io/projected/63e2dba6-3484-4b3f-b768-18afb682b15c-kube-api-access-9twpf\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.105357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.105347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e2dba6-3484-4b3f-b768-18afb682b15c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.122789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.122760    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b88cc7879-lhdgt"]
Sep 07 10:21:04.141673 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.141653    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b88cc7879-lhdgt"]
Sep 07 10:21:04.141673 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.141665    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206192 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-tls\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-ca-trust-extracted\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-bound-sa-token\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/763acf38-09b1-446d-b660-edbae66af565-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.206347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gww5c\" (UniqueName: \"kubernetes.io/projected/763acf38-09b1-446d-b660-edbae66af565-kube-api-access-gww5c\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206356    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-trusted-ca\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206458    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8059065a-ceda-4990-a33c-476e8c5d4300-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763acf38-09b1-446d-b660-edbae66af565-tmp\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.206553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwn2\" (UniqueName: \"kubernetes.io/projected/8059065a-ceda-4990-a33c-476e8c5d4300-kube-api-access-njwn2\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.206882 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twpf\" (UniqueName: \"kubernetes.io/projected/63e2dba6-3484-4b3f-b768-18afb682b15c-kube-api-access-9twpf\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.206882 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-certificates\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.206882 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.206882 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e2dba6-3484-4b3f-b768-18afb682b15c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.207063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.206886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4qw\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-kube-api-access-vj4qw\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.207201 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.207180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/763acf38-09b1-446d-b660-edbae66af565-tmp\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.207265 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.207227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-image-registry-private-configuration\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.207347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.207275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-installation-pull-secrets\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.207588 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.207545    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/63e2dba6-3484-4b3f-b768-18afb682b15c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.209656 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.209633    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-ca\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.209911 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.209894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/763acf38-09b1-446d-b660-edbae66af565-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.210490 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.210470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.210490 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.210481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8059065a-ceda-4990-a33c-476e8c5d4300-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.210631 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.210481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-hub\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.210849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.210827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/63e2dba6-3484-4b3f-b768-18afb682b15c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.219147 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.219037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gww5c\" (UniqueName: \"kubernetes.io/projected/763acf38-09b1-446d-b660-edbae66af565-kube-api-access-gww5c\") pod \"klusterlet-addon-workmgr-7b6f9c69bb-44clz\" (UID: \"763acf38-09b1-446d-b660-edbae66af565\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.219924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.219882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwn2\" (UniqueName: \"kubernetes.io/projected/8059065a-ceda-4990-a33c-476e8c5d4300-kube-api-access-njwn2\") pod \"managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp\" (UID: \"8059065a-ceda-4990-a33c-476e8c5d4300\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.220348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.220328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twpf\" (UniqueName: \"kubernetes.io/projected/63e2dba6-3484-4b3f-b768-18afb682b15c-kube-api-access-9twpf\") pod \"cluster-proxy-proxy-agent-fd75b87f-tkxbh\" (UID: \"63e2dba6-3484-4b3f-b768-18afb682b15c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.308025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.307954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-trusted-ca\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-certificates\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4qw\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-kube-api-access-vj4qw\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-image-registry-private-configuration\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-installation-pull-secrets\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-tls\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308194    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-ca-trust-extracted\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-bound-sa-token\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.308893 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-ca-trust-extracted\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.309014 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.308936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-certificates\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.309071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.309039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-trusted-ca\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.311051 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.311023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-installation-pull-secrets\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.311350 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.311331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-image-registry-private-configuration\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.311455 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.311442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-registry-tls\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.322200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.322178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-bound-sa-token\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.322424 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.322409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4qw\" (UniqueName: \"kubernetes.io/projected/0e939e49-8cbe-46f9-8000-6c5d0b68f78c-kube-api-access-vj4qw\") pod \"image-registry-6b88cc7879-lhdgt\" (UID: \"0e939e49-8cbe-46f9-8000-6c5d0b68f78c\") " pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.357866 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.357840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"
Sep 07 10:21:04.368539 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.368511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:04.385756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.385706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"
Sep 07 10:21:04.468669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.468635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:04.717797 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.717763    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sd5vw" event={"ID":"9991cffd-bfa1-4c64-8e07-4800532d94e4","Type":"ContainerStarted","Data":"302474dde8017ed687e3a64278c5afd499b66167c50ccdd38103c9918276e9ec"}
Sep 07 10:21:04.833702 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.833671    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"]
Sep 07 10:21:04.840480 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.840454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp"]
Sep 07 10:21:04.842357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.842333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh"]
Sep 07 10:21:04.843390 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:04.843369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b88cc7879-lhdgt"]
Sep 07 10:21:04.861925 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:04.861902    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763acf38_09b1_446d_b660_edbae66af565.slice/crio-3a91211ff3c01ce2e0126ab463b27753add4fd09cafac56f8a94182286ee49ea WatchSource:0}: Error finding container 3a91211ff3c01ce2e0126ab463b27753add4fd09cafac56f8a94182286ee49ea: Status 404 returned error can't find the container with id 3a91211ff3c01ce2e0126ab463b27753add4fd09cafac56f8a94182286ee49ea
Sep 07 10:21:04.862661 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:04.862633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63e2dba6_3484_4b3f_b768_18afb682b15c.slice/crio-16b02a3e61dba698968dd387b6d83725e47575ae84cdb0900e24a011d43c8324 WatchSource:0}: Error finding container 16b02a3e61dba698968dd387b6d83725e47575ae84cdb0900e24a011d43c8324: Status 404 returned error can't find the container with id 16b02a3e61dba698968dd387b6d83725e47575ae84cdb0900e24a011d43c8324
Sep 07 10:21:04.864502 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:04.863442    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e939e49_8cbe_46f9_8000_6c5d0b68f78c.slice/crio-0d9bd9f47960587284faa841fe07fd11ce89f1283b2015c1a1efde1a49843b2e WatchSource:0}: Error finding container 0d9bd9f47960587284faa841fe07fd11ce89f1283b2015c1a1efde1a49843b2e: Status 404 returned error can't find the container with id 0d9bd9f47960587284faa841fe07fd11ce89f1283b2015c1a1efde1a49843b2e
Sep 07 10:21:04.864502 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:04.864355    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8059065a_ceda_4990_a33c_476e8c5d4300.slice/crio-a6041778cb38ddd2b72e0d89c54ffd2590d70bb6f542a568d72919f35b3bc283 WatchSource:0}: Error finding container a6041778cb38ddd2b72e0d89c54ffd2590d70bb6f542a568d72919f35b3bc283: Status 404 returned error can't find the container with id a6041778cb38ddd2b72e0d89c54ffd2590d70bb6f542a568d72919f35b3bc283
Sep 07 10:21:05.721943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.721904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh" event={"ID":"63e2dba6-3484-4b3f-b768-18afb682b15c","Type":"ContainerStarted","Data":"16b02a3e61dba698968dd387b6d83725e47575ae84cdb0900e24a011d43c8324"}
Sep 07 10:21:05.724057 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.724026    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6m9dk" event={"ID":"245a647f-a221-48f1-8175-37626932d896","Type":"ContainerStarted","Data":"236dbbf10d9722a7116e8478598880fe26c80daf9dc4ac0d3883315ba7e7207e"}
Sep 07 10:21:05.724160 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.724121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6m9dk" event={"ID":"245a647f-a221-48f1-8175-37626932d896","Type":"ContainerStarted","Data":"00da413aa0f009945ae6e17fcff7f9ba45b9fb5099a10a0fbb09d6940ef57bff"}
Sep 07 10:21:05.724322 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.724257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:21:05.725378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.725355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp" event={"ID":"8059065a-ceda-4990-a33c-476e8c5d4300","Type":"ContainerStarted","Data":"a6041778cb38ddd2b72e0d89c54ffd2590d70bb6f542a568d72919f35b3bc283"}
Sep 07 10:21:05.726908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.726875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt" event={"ID":"0e939e49-8cbe-46f9-8000-6c5d0b68f78c","Type":"ContainerStarted","Data":"30741a6617899db5f6c7b2671ee47cececb246cbf88bdc35e17473eaa01ae581"}
Sep 07 10:21:05.727003 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.726928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt" event={"ID":"0e939e49-8cbe-46f9-8000-6c5d0b68f78c","Type":"ContainerStarted","Data":"0d9bd9f47960587284faa841fe07fd11ce89f1283b2015c1a1efde1a49843b2e"}
Sep 07 10:21:05.727061 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.727026    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:05.728516 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.728500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz" event={"ID":"763acf38-09b1-446d-b660-edbae66af565","Type":"ContainerStarted","Data":"3a91211ff3c01ce2e0126ab463b27753add4fd09cafac56f8a94182286ee49ea"}
Sep 07 10:21:05.745013 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.744969    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6m9dk" podStartSLOduration=17.397402406 podStartE2EDuration="19.744955748s" totalImagesPullingTime="2.347553342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:05.744158626 +0000 UTC m=+53.754811942" watchObservedRunningTime="2026-09-07 10:21:05.744955748 +0000 UTC m=+53.755609063"
Sep 07 10:21:05.770285 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:05.770235    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt" podStartSLOduration=1.770217741 podStartE2EDuration="1.770217741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:05.769922539 +0000 UTC m=+53.780575853" watchObservedRunningTime="2026-09-07 10:21:05.770217741 +0000 UTC m=+53.780871054"
Sep 07 10:21:06.738621 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:06.738124    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sd5vw" event={"ID":"9991cffd-bfa1-4c64-8e07-4800532d94e4","Type":"ContainerStarted","Data":"932dea40f90430d66c630258169737ceb6fa654b038907ae7b82f803e7e72003"}
Sep 07 10:21:06.756852 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:06.756656    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sd5vw" podStartSLOduration=17.765126267 podStartE2EDuration="20.756635636s" totalImagesPullingTime="2.991509369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:06.754889899 +0000 UTC m=+54.765543211" watchObservedRunningTime="2026-09-07 10:21:06.756635636 +0000 UTC m=+54.767288951"
Sep 07 10:21:11.657812 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:11.657785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k52cb"
Sep 07 10:21:12.760219 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.760183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz" event={"ID":"763acf38-09b1-446d-b660-edbae66af565","Type":"ContainerStarted","Data":"295301057a84af92a25f7f8e392ad364535be24fe026e4efc629b7446beaa59e"}
Sep 07 10:21:12.760629 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.760341    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:12.762181 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.762161    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz"
Sep 07 10:21:12.762588 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.762567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh" event={"ID":"63e2dba6-3484-4b3f-b768-18afb682b15c","Type":"ContainerStarted","Data":"eb69db1c49ba8a4a804ad12a4cfadd57c09e1b41ce5bc4303fe262200070b0a8"}
Sep 07 10:21:12.762655 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.762599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh" event={"ID":"63e2dba6-3484-4b3f-b768-18afb682b15c","Type":"ContainerStarted","Data":"60340ce650d73145c56d66b3e2e84d41c63e3a519aca9fbd5463cc47d47e97d9"}
Sep 07 10:21:12.762655 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.762616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh" event={"ID":"63e2dba6-3484-4b3f-b768-18afb682b15c","Type":"ContainerStarted","Data":"9b3ceef38e8fe76cbb058dfebf0b61182e7001d5d22ee8d16a394c4cc0588f8e"}
Sep 07 10:21:12.764148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.764128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp" event={"ID":"8059065a-ceda-4990-a33c-476e8c5d4300","Type":"ContainerStarted","Data":"64ca0b78c61b5843ed473c25bc5330b5c6026bba1896bf4bd1f2d9791083e4e9"}
Sep 07 10:21:12.776677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.776632    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b6f9c69bb-44clz" podStartSLOduration=2.807605667 podStartE2EDuration="9.77661887s" totalImagesPullingTime="6.969013203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:12.775978715 +0000 UTC m=+60.786632028" watchObservedRunningTime="2026-09-07 10:21:12.77661887 +0000 UTC m=+60.787272181"
Sep 07 10:21:12.793449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.793406    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-fd75b87f-tkxbh" podStartSLOduration=2.810711358 podStartE2EDuration="9.793395055s" totalImagesPullingTime="6.982683697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:12.792364231 +0000 UTC m=+60.803017545" watchObservedRunningTime="2026-09-07 10:21:12.793395055 +0000 UTC m=+60.804048375"
Sep 07 10:21:12.824147 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:12.824106    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c5dfb669b-mdpfp" podStartSLOduration=2.857003836 podStartE2EDuration="9.824094143s" totalImagesPullingTime="6.967090307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:12.823322291 +0000 UTC m=+60.833975602" watchObservedRunningTime="2026-09-07 10:21:12.824094143 +0000 UTC m=+60.834747456"
Sep 07 10:21:15.739272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:15.739239    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6m9dk"
Sep 07 10:21:18.324503 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.324467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:21:18.324503 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.324504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:21:18.327500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.327466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:21:18.327635 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.327616    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:21:18.338013 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.337975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d824971e-4bb2-4633-b3b6-f705d282054c-original-pull-secret\") pod \"global-pull-secret-syncer-tc82s\" (UID: \"d824971e-4bb2-4633-b3b6-f705d282054c\") " pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:21:18.338013 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.337992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e92996d-bebd-40a6-9f37-771ea7981a16-metrics-certs\") pod \"network-metrics-daemon-zhvfh\" (UID: \"3e92996d-bebd-40a6-9f37-771ea7981a16\") " pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:21:18.400352 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.400325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tc82s"
Sep 07 10:21:18.411690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.411667    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbx74\""
Sep 07 10:21:18.419789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.419776    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zhvfh"
Sep 07 10:21:18.425143 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.425127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:21:18.427497 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.427484    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:21:18.437944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.437926    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:21:18.448260 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.448221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfqf2\" (UniqueName: \"kubernetes.io/projected/dee58893-1ba9-408d-8e4d-1c5853c16a5d-kube-api-access-jfqf2\") pod \"network-check-target-9n2kt\" (UID: \"dee58893-1ba9-408d-8e4d-1c5853c16a5d\") " pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:21:18.702497 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.702370    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tc82s"]
Sep 07 10:21:18.704990 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:18.704967    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd824971e_4bb2_4633_b3b6_f705d282054c.slice/crio-d4fff6616fc64eaaea1a2a177eaa46ed96df37a71d7cd506cfd2603bfe28978d WatchSource:0}: Error finding container d4fff6616fc64eaaea1a2a177eaa46ed96df37a71d7cd506cfd2603bfe28978d: Status 404 returned error can't find the container with id d4fff6616fc64eaaea1a2a177eaa46ed96df37a71d7cd506cfd2603bfe28978d
Sep 07 10:21:18.707058 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.707027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jrgm\""
Sep 07 10:21:18.715503 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.715485    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:21:18.723629 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.723610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zhvfh"]
Sep 07 10:21:18.725970 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:18.725946    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e92996d_bebd_40a6_9f37_771ea7981a16.slice/crio-1ec930f9a0d7e441fff1583067293d38d38586ca07d65c91cbcbc4b6f3c7caff WatchSource:0}: Error finding container 1ec930f9a0d7e441fff1583067293d38d38586ca07d65c91cbcbc4b6f3c7caff: Status 404 returned error can't find the container with id 1ec930f9a0d7e441fff1583067293d38d38586ca07d65c91cbcbc4b6f3c7caff
Sep 07 10:21:18.784931 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.784893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zhvfh" event={"ID":"3e92996d-bebd-40a6-9f37-771ea7981a16","Type":"ContainerStarted","Data":"1ec930f9a0d7e441fff1583067293d38d38586ca07d65c91cbcbc4b6f3c7caff"}
Sep 07 10:21:18.786123 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:18.786092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tc82s" event={"ID":"d824971e-4bb2-4633-b3b6-f705d282054c","Type":"ContainerStarted","Data":"d4fff6616fc64eaaea1a2a177eaa46ed96df37a71d7cd506cfd2603bfe28978d"}
Sep 07 10:21:19.009394 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:19.009324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9n2kt"]
Sep 07 10:21:19.012185 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:21:19.012159    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee58893_1ba9_408d_8e4d_1c5853c16a5d.slice/crio-a38b7ea9dff4347cff0dea6b41867eb3556c62e8ad360ae55ee8f1c257edb19a WatchSource:0}: Error finding container a38b7ea9dff4347cff0dea6b41867eb3556c62e8ad360ae55ee8f1c257edb19a: Status 404 returned error can't find the container with id a38b7ea9dff4347cff0dea6b41867eb3556c62e8ad360ae55ee8f1c257edb19a
Sep 07 10:21:19.792421 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:19.792389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9n2kt" event={"ID":"dee58893-1ba9-408d-8e4d-1c5853c16a5d","Type":"ContainerStarted","Data":"a38b7ea9dff4347cff0dea6b41867eb3556c62e8ad360ae55ee8f1c257edb19a"}
Sep 07 10:21:20.520615 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.520554    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5tgwg"]
Sep 07 10:21:20.554166 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.554141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.556943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.556915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 10:21:20.557160 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.557143    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 10:21:20.557659 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.557641    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 10:21:20.557853 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.557824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8f4ct\""
Sep 07 10:21:20.558240 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.558221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 10:21:20.558914 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.558895    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 10:21:20.559112 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.559095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-root\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-textfile\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qs6c\" (UniqueName: \"kubernetes.io/projected/ee3e0294-8df7-4731-ac8c-ef5c025134a4-kube-api-access-2qs6c\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-sys\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-wtmp\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-tls\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-metrics-client-ca\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.644724 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.644574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.745876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-sys\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.745920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-wtmp\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.745950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-tls\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.745980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-metrics-client-ca\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-root\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-textfile\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs6c\" (UniqueName: \"kubernetes.io/projected/ee3e0294-8df7-4731-ac8c-ef5c025134a4-kube-api-access-2qs6c\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746519    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-sys\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.746645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-wtmp\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.756622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.756456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee3e0294-8df7-4731-ac8c-ef5c025134a4-root\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.757343 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.756933    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-metrics-client-ca\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.758046 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.757409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-accelerators-collector-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.758046 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.757704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-textfile\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.758046 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.758007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-tls\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.759723 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.759702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qs6c\" (UniqueName: \"kubernetes.io/projected/ee3e0294-8df7-4731-ac8c-ef5c025134a4-kube-api-access-2qs6c\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.768048 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.767996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee3e0294-8df7-4731-ac8c-ef5c025134a4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5tgwg\" (UID: \"ee3e0294-8df7-4731-ac8c-ef5c025134a4\") " pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:20.800988 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.800209    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zhvfh" event={"ID":"3e92996d-bebd-40a6-9f37-771ea7981a16","Type":"ContainerStarted","Data":"a772be596bb1fa53b2d9dfd181fea1c6416a443c7c558ac006487853f87f3c76"}
Sep 07 10:21:20.800988 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.800270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zhvfh" event={"ID":"3e92996d-bebd-40a6-9f37-771ea7981a16","Type":"ContainerStarted","Data":"ea2934cbe85ff07a5bf2509391d041d7eabd5f047c27753abadedb6f4af267bb"}
Sep 07 10:21:20.817379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.817331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zhvfh" podStartSLOduration=67.289071469 podStartE2EDuration="1m8.8173145s" totalImagesPullingTime="1.528243031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:20.817181984 +0000 UTC m=+68.827835299" watchObservedRunningTime="2026-09-07 10:21:20.8173145 +0000 UTC m=+68.827967806"
Sep 07 10:21:20.865714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:20.865539    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5tgwg"
Sep 07 10:21:21.806020 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:21.805978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tgwg" event={"ID":"ee3e0294-8df7-4731-ac8c-ef5c025134a4","Type":"ContainerStarted","Data":"f4b5aecc6b66d0d763beed90a435755238ea5a39b7a2e7e051cbc714e22acdd6"}
Sep 07 10:21:24.472885 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.472803    2562 patch_prober.go:28] interesting pod/image-registry-6b88cc7879-lhdgt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 07 10:21:24.472885 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.472855    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt" podUID="0e939e49-8cbe-46f9-8000-6c5d0b68f78c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 10:21:24.723803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.723733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:24.818669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.818640    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee3e0294-8df7-4731-ac8c-ef5c025134a4" containerID="f88cc783b7f1fc0388144f2423ee4bd0b2e471cb6970f019289ceadaf3797c63" exitCode=0
Sep 07 10:21:24.818832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.818721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tgwg" event={"ID":"ee3e0294-8df7-4731-ac8c-ef5c025134a4","Type":"ContainerDied","Data":"f88cc783b7f1fc0388144f2423ee4bd0b2e471cb6970f019289ceadaf3797c63"}
Sep 07 10:21:24.820360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.820338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tc82s" event={"ID":"d824971e-4bb2-4633-b3b6-f705d282054c","Type":"ContainerStarted","Data":"c9aa055c396ce9c638e55c6c5aa6d3194f3b46dfdec36e796b69b98e123c8f7d"}
Sep 07 10:21:24.821935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.821914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9n2kt" event={"ID":"dee58893-1ba9-408d-8e4d-1c5853c16a5d","Type":"ContainerStarted","Data":"0504d72a21977f0487b6529494740abe93bea837fd0d77573752a1c29ff5b456"}
Sep 07 10:21:24.822032 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.821969    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:21:24.852386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.851775    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9n2kt" podStartSLOduration=67.830313054 podStartE2EDuration="1m12.851759171s" totalImagesPullingTime="5.021446117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:24.851227208 +0000 UTC m=+72.861880518" watchObservedRunningTime="2026-09-07 10:21:24.851759171 +0000 UTC m=+72.862412485"
Sep 07 10:21:24.866179 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:24.866141    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tc82s" podStartSLOduration=67.537447075 podStartE2EDuration="1m12.866128884s" totalImagesPullingTime="5.328681809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:24.865478998 +0000 UTC m=+72.876132312" watchObservedRunningTime="2026-09-07 10:21:24.866128884 +0000 UTC m=+72.876782209"
Sep 07 10:21:25.827750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:25.827690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tgwg" event={"ID":"ee3e0294-8df7-4731-ac8c-ef5c025134a4","Type":"ContainerStarted","Data":"bd65de5a9e858c9d939ea81ac3ac83ccf247ed56107f9d96601018319f8e6b19"}
Sep 07 10:21:25.827750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:25.827751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5tgwg" event={"ID":"ee3e0294-8df7-4731-ac8c-ef5c025134a4","Type":"ContainerStarted","Data":"6150f8bec3a4f98599df661d398ec576480e08c5109dd2e64a81c1198cc1a4c1"}
Sep 07 10:21:25.846577 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:25.846529    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5tgwg" podStartSLOduration=2.604557024 podStartE2EDuration="5.846516951s" totalImagesPullingTime="3.1742486s" totalInitContainerRuntime="67.711327ms" isStatefulPod=true podCreationTimestamp="2026-09-07 10:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:25.845178534 +0000 UTC m=+73.855831847" watchObservedRunningTime="2026-09-07 10:21:25.846516951 +0000 UTC m=+73.857170263"
Sep 07 10:21:26.742811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:26.742785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b88cc7879-lhdgt"
Sep 07 10:21:29.736343 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:29.736279    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" podUID="4a1827ec-75e6-47bc-baf6-3e102837195f" containerName="registry" containerID="cri-o://5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428" gracePeriod=30
Sep 07 10:21:29.969158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:29.969133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:30.114650 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114626    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.114823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.114823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkglt\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.114823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114701    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.114823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114731    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.114823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114790    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.115136 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.115136 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.114885    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token\") pod \"4a1827ec-75e6-47bc-baf6-3e102837195f\" (UID: \"4a1827ec-75e6-47bc-baf6-3e102837195f\") "
Sep 07 10:21:30.115276 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.115246    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:21:30.115464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.115414    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:21:30.117403 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.117360    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt" (OuterVolumeSpecName: "kube-api-access-fkglt") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "kube-api-access-fkglt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:21:30.117514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.117402    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:21:30.117514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.117483    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:21:30.117660 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.117643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:21:30.117731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.117715    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:21:30.123359 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.123293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4a1827ec-75e6-47bc-baf6-3e102837195f" (UID: "4a1827ec-75e6-47bc-baf6-3e102837195f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:21:30.215640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215620    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a1827ec-75e6-47bc-baf6-3e102837195f-ca-trust-extracted\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215643    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-installation-pull-secrets\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215656    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkglt\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-kube-api-access-fkglt\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215670    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a1827ec-75e6-47bc-baf6-3e102837195f-image-registry-private-configuration\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215684    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-tls\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215695    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-trusted-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215707    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a1827ec-75e6-47bc-baf6-3e102837195f-registry-certificates\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.215741 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.215719    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a1827ec-75e6-47bc-baf6-3e102837195f-bound-sa-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:21:30.847053 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.847019    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a1827ec-75e6-47bc-baf6-3e102837195f" containerID="5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428" exitCode=0
Sep 07 10:21:30.847475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.847065    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48"
Sep 07 10:21:30.847475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.847112    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" event={"ID":"4a1827ec-75e6-47bc-baf6-3e102837195f","Type":"ContainerDied","Data":"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"}
Sep 07 10:21:30.847475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.847166    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b47bc4647-ghs48" event={"ID":"4a1827ec-75e6-47bc-baf6-3e102837195f","Type":"ContainerDied","Data":"e1f572cac3cb28414c865ec88ca22c013b8081208994bc2bbabb1cffe1a87d8d"}
Sep 07 10:21:30.847475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.847187    2562 scope.go:117] "RemoveContainer" containerID="5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"
Sep 07 10:21:30.855105 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.855087    2562 scope.go:117] "RemoveContainer" containerID="5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"
Sep 07 10:21:30.855397 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:21:30.855369    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428\": container with ID starting with 5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428 not found: ID does not exist" containerID="5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"
Sep 07 10:21:30.855488 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.855405    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428"} err="failed to get container status \"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428\": rpc error: code = NotFound desc = could not find container \"5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428\": container with ID starting with 5797b995e4003caaa0ffed8edf0a29bce1998b14f5e60c2906338e0f8eaf1428 not found: ID does not exist"
Sep 07 10:21:30.863807 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.863784    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:21:30.869468 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:30.869450    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b47bc4647-ghs48"]
Sep 07 10:21:32.493968 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:32.493934    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1827ec-75e6-47bc-baf6-3e102837195f" path="/var/lib/kubelet/pods/4a1827ec-75e6-47bc-baf6-3e102837195f/volumes"
Sep 07 10:21:55.830177 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:21:55.830145    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9n2kt"
Sep 07 10:25:12.396666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:25:12.396635    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:25:12.401159 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:25:12.401139    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:25:12.402965 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:25:12.402946    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 10:30:12.421984 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:30:12.421955    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:30:12.422916 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:30:12.422890    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:31:36.152773 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.152745    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"]
Sep 07 10:31:36.153218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.152964    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1827ec-75e6-47bc-baf6-3e102837195f" containerName="registry"
Sep 07 10:31:36.153218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.152974    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1827ec-75e6-47bc-baf6-3e102837195f" containerName="registry"
Sep 07 10:31:36.153218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.153013    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1827ec-75e6-47bc-baf6-3e102837195f" containerName="registry"
Sep 07 10:31:36.155893 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.155872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.165611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.165591    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 10:31:36.166480 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.166462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 10:31:36.166562 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.166465    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 10:31:36.166650 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.166467    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t7dj9\""
Sep 07 10:31:36.167079 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.167055    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 10:31:36.198735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.198707    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"]
Sep 07 10:31:36.212289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.212269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.212394 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.212311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.212394 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.212337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.212458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.212391    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rks6t\" (UniqueName: \"kubernetes.io/projected/426f9824-2a3f-4118-9d9f-4c7852c8462f-kube-api-access-rks6t\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.212458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.212434    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313423 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313522 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313522 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rks6t\" (UniqueName: \"kubernetes.io/projected/426f9824-2a3f-4118-9d9f-4c7852c8462f-kube-api-access-rks6t\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313522 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.313844 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.313826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.315749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.315722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/426f9824-2a3f-4118-9d9f-4c7852c8462f-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.316022 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.316003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.316084 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.316024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/426f9824-2a3f-4118-9d9f-4c7852c8462f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.327254 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.327232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rks6t\" (UniqueName: \"kubernetes.io/projected/426f9824-2a3f-4118-9d9f-4c7852c8462f-kube-api-access-rks6t\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-6tv7t\" (UID: \"426f9824-2a3f-4118-9d9f-4c7852c8462f\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.465281 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.465211    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:36.745530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.745509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"]
Sep 07 10:31:36.747101 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:31:36.747075    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426f9824_2a3f_4118_9d9f_4c7852c8462f.slice/crio-cbd0a09dd24f52bfd9b6a1b44ded627137aa2292aedafe5eb2cad9e6cfebde67 WatchSource:0}: Error finding container cbd0a09dd24f52bfd9b6a1b44ded627137aa2292aedafe5eb2cad9e6cfebde67: Status 404 returned error can't find the container with id cbd0a09dd24f52bfd9b6a1b44ded627137aa2292aedafe5eb2cad9e6cfebde67
Sep 07 10:31:36.748881 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.748865    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:31:36.776178 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:36.776146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t" event={"ID":"426f9824-2a3f-4118-9d9f-4c7852c8462f","Type":"ContainerStarted","Data":"cbd0a09dd24f52bfd9b6a1b44ded627137aa2292aedafe5eb2cad9e6cfebde67"}
Sep 07 10:31:39.789048 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:39.789019    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t" event={"ID":"426f9824-2a3f-4118-9d9f-4c7852c8462f","Type":"ContainerStarted","Data":"05f8cfbe1a96a8d541e8a81dbc35caba7d80493e21538ec60e2a8271973bf4a5"}
Sep 07 10:31:40.794942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:40.794908    2562 generic.go:356] "Generic (PLEG): container finished" podID="426f9824-2a3f-4118-9d9f-4c7852c8462f" containerID="05f8cfbe1a96a8d541e8a81dbc35caba7d80493e21538ec60e2a8271973bf4a5" exitCode=0
Sep 07 10:31:40.795267 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:40.794956    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t" event={"ID":"426f9824-2a3f-4118-9d9f-4c7852c8462f","Type":"ContainerDied","Data":"05f8cfbe1a96a8d541e8a81dbc35caba7d80493e21538ec60e2a8271973bf4a5"}
Sep 07 10:31:41.800068 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:41.800030    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t" event={"ID":"426f9824-2a3f-4118-9d9f-4c7852c8462f","Type":"ContainerStarted","Data":"f43562a6517195039b7296c8feb246f5563f0e95054ecaaf3b4bfbf036b1fe89"}
Sep 07 10:31:41.800529 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:41.800178    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:41.830842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:41.830793    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t" podStartSLOduration=1.696422702 podStartE2EDuration="5.830778369s" totalImagesPullingTime="3.200341582s" totalInitContainerRuntime="934.014085ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:41.828694077 +0000 UTC m=+689.839347392" watchObservedRunningTime="2026-09-07 10:31:41.830778369 +0000 UTC m=+689.841431683"
Sep 07 10:31:52.805664 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:52.805634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-6tv7t"
Sep 07 10:31:55.864640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.864609    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7794764676-c68qd"]
Sep 07 10:31:55.873415 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.873391    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:55.876433 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.876402    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sc5hq\""
Sep 07 10:31:55.877631 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.877608    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 10:31:55.877739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.877638    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 10:31:55.877804 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.877763    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 10:31:55.877893 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.877874    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 10:31:55.885814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.885793    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7794764676-c68qd"]
Sep 07 10:31:55.948840 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.948802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tls-certs\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:55.949012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.948870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltbp\" (UniqueName: \"kubernetes.io/projected/c6c8671f-5ef3-421f-a99b-e4c02bf46504-kube-api-access-mltbp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:55.949012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:55.948899    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tmp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.049880 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.049843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltbp\" (UniqueName: \"kubernetes.io/projected/c6c8671f-5ef3-421f-a99b-e4c02bf46504-kube-api-access-mltbp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.050064 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.049879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tmp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.050064 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.049937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tls-certs\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.052369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.052345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tmp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.052485 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.052460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c8671f-5ef3-421f-a99b-e4c02bf46504-tls-certs\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.060322 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.060284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltbp\" (UniqueName: \"kubernetes.io/projected/c6c8671f-5ef3-421f-a99b-e4c02bf46504-kube-api-access-mltbp\") pod \"kube-auth-proxy-7794764676-c68qd\" (UID: \"c6c8671f-5ef3-421f-a99b-e4c02bf46504\") " pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.182572 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.182495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd"
Sep 07 10:31:56.463270 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.463202    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7794764676-c68qd"]
Sep 07 10:31:56.466063 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:31:56.466016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c8671f_5ef3_421f_a99b_e4c02bf46504.slice/crio-0cb7d4596d2ff88bf1845ad35a045d339be5d49906e218ed7a7525d9611e6a13 WatchSource:0}: Error finding container 0cb7d4596d2ff88bf1845ad35a045d339be5d49906e218ed7a7525d9611e6a13: Status 404 returned error can't find the container with id 0cb7d4596d2ff88bf1845ad35a045d339be5d49906e218ed7a7525d9611e6a13
Sep 07 10:31:56.854886 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:56.854853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd" event={"ID":"c6c8671f-5ef3-421f-a99b-e4c02bf46504","Type":"ContainerStarted","Data":"0cb7d4596d2ff88bf1845ad35a045d339be5d49906e218ed7a7525d9611e6a13"}
Sep 07 10:31:59.866975 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:59.866940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd" event={"ID":"c6c8671f-5ef3-421f-a99b-e4c02bf46504","Type":"ContainerStarted","Data":"c5aed0e78f646fbb2df5516f311df3e17595de80b253860559c2a5d92cb5e263"}
Sep 07 10:31:59.885838 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:31:59.885788    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7794764676-c68qd" podStartSLOduration=2.183902983 podStartE2EDuration="4.885773524s" totalImagesPullingTime="2.701870541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:59.884981809 +0000 UTC m=+707.895635125" watchObservedRunningTime="2026-09-07 10:31:59.885773524 +0000 UTC m=+707.896426868"
Sep 07 10:32:01.064372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.064334    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"]
Sep 07 10:32:01.066402 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.066380    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.070126 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070098    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 10:32:01.070126 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070105    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qgd92\""
Sep 07 10:32:01.070336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:32:01.070336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070239    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 10:32:01.070336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070258    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 10:32:01.070475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.070353    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 10:32:01.077982 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.077962    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"]
Sep 07 10:32:01.194339 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.194293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-manager-config\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.194451 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.194344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-metrics-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.194451 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.194401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.194525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.194473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4tg\" (UniqueName: \"kubernetes.io/projected/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-kube-api-access-mv4tg\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.295356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.295319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4tg\" (UniqueName: \"kubernetes.io/projected/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-kube-api-access-mv4tg\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.295516 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.295363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-manager-config\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.295516 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.295384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-metrics-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.295516 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.295413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.296136 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.296060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-manager-config\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.297996 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.297979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-metrics-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.298084 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.298071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-cert\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.304479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.304458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4tg\" (UniqueName: \"kubernetes.io/projected/34b7c087-0faf-4a41-8b99-9c3935f2c8a6-kube-api-access-mv4tg\") pod \"lws-controller-manager-547fb475d5-5z9lr\" (UID: \"34b7c087-0faf-4a41-8b99-9c3935f2c8a6\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.375802 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.375717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:01.643475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.643406    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"]
Sep 07 10:32:01.646232 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:32:01.646209    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b7c087_0faf_4a41_8b99_9c3935f2c8a6.slice/crio-59c258fdc3b92c8ee5dde32948bea3f0a5a98fb623c0d82d97e6a7bb36992ebb WatchSource:0}: Error finding container 59c258fdc3b92c8ee5dde32948bea3f0a5a98fb623c0d82d97e6a7bb36992ebb: Status 404 returned error can't find the container with id 59c258fdc3b92c8ee5dde32948bea3f0a5a98fb623c0d82d97e6a7bb36992ebb
Sep 07 10:32:01.873950 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:01.873917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr" event={"ID":"34b7c087-0faf-4a41-8b99-9c3935f2c8a6","Type":"ContainerStarted","Data":"59c258fdc3b92c8ee5dde32948bea3f0a5a98fb623c0d82d97e6a7bb36992ebb"}
Sep 07 10:32:05.890007 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:05.889971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr" event={"ID":"34b7c087-0faf-4a41-8b99-9c3935f2c8a6","Type":"ContainerStarted","Data":"2686a3f5b63e4c49a70178030ccc56d345b593f1a7886a1cc20f9d77442b6316"}
Sep 07 10:32:05.890405 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:05.890114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:32:05.907801 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:05.907742    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr" podStartSLOduration=1.369044631 podStartE2EDuration="4.907721701s" totalImagesPullingTime="3.53867707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:05.907571921 +0000 UTC m=+713.918225239" watchObservedRunningTime="2026-09-07 10:32:05.907721701 +0000 UTC m=+713.918375016"
Sep 07 10:32:16.896286 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:32:16.896258    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-5z9lr"
Sep 07 10:34:39.342030 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.341996    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:39.344942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.344927    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:39.348068 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.348040    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 10:34:39.349034 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.349009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 10:34:39.349173 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.349034    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4tp6s\""
Sep 07 10:34:39.353779 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.353758    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:39.396658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.396623    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktml\" (UniqueName: \"kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml\") pod \"authorino-b54df94b4-n49xl\" (UID: \"00d2991b-0799-4fc9-8df0-69414fd321aa\") " pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:39.497364 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.497331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktml\" (UniqueName: \"kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml\") pod \"authorino-b54df94b4-n49xl\" (UID: \"00d2991b-0799-4fc9-8df0-69414fd321aa\") " pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:39.505790 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.505767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktml\" (UniqueName: \"kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml\") pod \"authorino-b54df94b4-n49xl\" (UID: \"00d2991b-0799-4fc9-8df0-69414fd321aa\") " pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:39.574533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.574504    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:34:39.577579 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.577562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:34:39.585076 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.585055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:34:39.598012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.597966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2qh\" (UniqueName: \"kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh\") pod \"authorino-c548b5d7d-5wbg8\" (UID: \"b77b5a95-ce89-4b97-bcc3-262b6b1f6529\") " pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:34:39.654674 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.654651    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:39.698750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.698715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2qh\" (UniqueName: \"kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh\") pod \"authorino-c548b5d7d-5wbg8\" (UID: \"b77b5a95-ce89-4b97-bcc3-262b6b1f6529\") " pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:34:39.708459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.708430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2qh\" (UniqueName: \"kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh\") pod \"authorino-c548b5d7d-5wbg8\" (UID: \"b77b5a95-ce89-4b97-bcc3-262b6b1f6529\") " pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:34:39.886841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.886759    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:34:39.932000 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:39.931966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:40.164001 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:40.163973    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:34:40.166268 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:34:40.166244    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77b5a95_ce89_4b97_bcc3_262b6b1f6529.slice/crio-9ab305cdb3f66379c92e495e5b6bc79c379432f3766eea92e69f5dfe9bf2a30b WatchSource:0}: Error finding container 9ab305cdb3f66379c92e495e5b6bc79c379432f3766eea92e69f5dfe9bf2a30b: Status 404 returned error can't find the container with id 9ab305cdb3f66379c92e495e5b6bc79c379432f3766eea92e69f5dfe9bf2a30b
Sep 07 10:34:40.425717 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:40.425627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-n49xl" event={"ID":"00d2991b-0799-4fc9-8df0-69414fd321aa","Type":"ContainerStarted","Data":"8c5b661e4649d79a2511fbae620cbd01125916f29b0189c8b66b04d6090ea23a"}
Sep 07 10:34:40.426634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:40.426613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" event={"ID":"b77b5a95-ce89-4b97-bcc3-262b6b1f6529","Type":"ContainerStarted","Data":"9ab305cdb3f66379c92e495e5b6bc79c379432f3766eea92e69f5dfe9bf2a30b"}
Sep 07 10:34:43.440863 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:43.440831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-n49xl" event={"ID":"00d2991b-0799-4fc9-8df0-69414fd321aa","Type":"ContainerStarted","Data":"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"}
Sep 07 10:34:43.442431 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:43.442406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" event={"ID":"b77b5a95-ce89-4b97-bcc3-262b6b1f6529","Type":"ContainerStarted","Data":"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"}
Sep 07 10:34:43.457907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:43.457867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-n49xl" podStartSLOduration=1.662284624 podStartE2EDuration="4.457852305s" totalImagesPullingTime="2.795567681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:43.457288332 +0000 UTC m=+871.467941647" watchObservedRunningTime="2026-09-07 10:34:43.457852305 +0000 UTC m=+871.468505619"
Sep 07 10:34:43.473534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:43.473484    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" podStartSLOduration=1.902668592 podStartE2EDuration="4.473464525s" totalImagesPullingTime="2.570795933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:43.473277984 +0000 UTC m=+871.483931296" watchObservedRunningTime="2026-09-07 10:34:43.473464525 +0000 UTC m=+871.484117841"
Sep 07 10:34:43.499831 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:43.499804    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:45.450645 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:45.450600    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-n49xl" podUID="00d2991b-0799-4fc9-8df0-69414fd321aa" containerName="authorino" containerID="cri-o://3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1" gracePeriod=30
Sep 07 10:34:45.689032 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:45.689010    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:45.740975 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:45.740906    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktml\" (UniqueName: \"kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml\") pod \"00d2991b-0799-4fc9-8df0-69414fd321aa\" (UID: \"00d2991b-0799-4fc9-8df0-69414fd321aa\") "
Sep 07 10:34:45.743054 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:45.743028    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml" (OuterVolumeSpecName: "kube-api-access-gktml") pod "00d2991b-0799-4fc9-8df0-69414fd321aa" (UID: "00d2991b-0799-4fc9-8df0-69414fd321aa"). InnerVolumeSpecName "kube-api-access-gktml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:34:45.841657 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:45.841621    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktml\" (UniqueName: \"kubernetes.io/projected/00d2991b-0799-4fc9-8df0-69414fd321aa-kube-api-access-gktml\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:34:46.455040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.455004    2562 generic.go:356] "Generic (PLEG): container finished" podID="00d2991b-0799-4fc9-8df0-69414fd321aa" containerID="3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1" exitCode=0
Sep 07 10:34:46.455475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.455048    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-n49xl"
Sep 07 10:34:46.455475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.455090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-n49xl" event={"ID":"00d2991b-0799-4fc9-8df0-69414fd321aa","Type":"ContainerDied","Data":"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"}
Sep 07 10:34:46.455475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.455132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-n49xl" event={"ID":"00d2991b-0799-4fc9-8df0-69414fd321aa","Type":"ContainerDied","Data":"8c5b661e4649d79a2511fbae620cbd01125916f29b0189c8b66b04d6090ea23a"}
Sep 07 10:34:46.455475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.455153    2562 scope.go:117] "RemoveContainer" containerID="3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"
Sep 07 10:34:46.463863 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.463847    2562 scope.go:117] "RemoveContainer" containerID="3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"
Sep 07 10:34:46.464110 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:34:46.464092    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1\": container with ID starting with 3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1 not found: ID does not exist" containerID="3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"
Sep 07 10:34:46.464156 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.464121    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1"} err="failed to get container status \"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1\": rpc error: code = NotFound desc = could not find container \"3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1\": container with ID starting with 3cf01eab6712ff3e5a09fd593ebeadab34107f96b558e766bd2f2d5cbcc04ad1 not found: ID does not exist"
Sep 07 10:34:46.475769 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.475749    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:46.484413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.484396    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-n49xl"]
Sep 07 10:34:46.494436 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:46.494417    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d2991b-0799-4fc9-8df0-69414fd321aa" path="/var/lib/kubelet/pods/00d2991b-0799-4fc9-8df0-69414fd321aa/volumes"
Sep 07 10:34:56.574983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.574953    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"]
Sep 07 10:34:56.575489 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.575180    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d2991b-0799-4fc9-8df0-69414fd321aa" containerName="authorino"
Sep 07 10:34:56.575489 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.575191    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d2991b-0799-4fc9-8df0-69414fd321aa" containerName="authorino"
Sep 07 10:34:56.575489 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.575242    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d2991b-0799-4fc9-8df0-69414fd321aa" containerName="authorino"
Sep 07 10:34:56.580316 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.580279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.583702 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.583680    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 07 10:34:56.583971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.583955    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rhhmh\""
Sep 07 10:34:56.584816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.584799    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 10:34:56.584912 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.584877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:34:56.588083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.588066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"]
Sep 07 10:34:56.613576 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.613546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-data\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.613676 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.613582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx6db\" (UniqueName: \"kubernetes.io/projected/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-kube-api-access-sx6db\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.714860 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.714832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-data\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.714950 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.714863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx6db\" (UniqueName: \"kubernetes.io/projected/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-kube-api-access-sx6db\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.715161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.715143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-data\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.723272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.723251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx6db\" (UniqueName: \"kubernetes.io/projected/d9f78643-8675-42a2-8aa0-d7e4770ab1cf-kube-api-access-sx6db\") pod \"postgres-69fb4594cf-22kwg\" (UID: \"d9f78643-8675-42a2-8aa0-d7e4770ab1cf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:56.892439 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:56.892379    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:34:57.165469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:57.165446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"]
Sep 07 10:34:57.167934 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:34:57.167911    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f78643_8675_42a2_8aa0_d7e4770ab1cf.slice/crio-9ff0b6e967b21fb48c5da33e692a9d63f2087e68fe541bc40234c266e3ca0bea WatchSource:0}: Error finding container 9ff0b6e967b21fb48c5da33e692a9d63f2087e68fe541bc40234c266e3ca0bea: Status 404 returned error can't find the container with id 9ff0b6e967b21fb48c5da33e692a9d63f2087e68fe541bc40234c266e3ca0bea
Sep 07 10:34:57.493670 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:34:57.493596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg" event={"ID":"d9f78643-8675-42a2-8aa0-d7e4770ab1cf","Type":"ContainerStarted","Data":"9ff0b6e967b21fb48c5da33e692a9d63f2087e68fe541bc40234c266e3ca0bea"}
Sep 07 10:35:02.515038 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:02.515003    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg" event={"ID":"d9f78643-8675-42a2-8aa0-d7e4770ab1cf","Type":"ContainerStarted","Data":"bc3af468eba74a858f1aa0fe74944bdac09e564ff62d654ddb94c90453fbf749"}
Sep 07 10:35:02.515397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:02.515066    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:35:02.532962 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:02.532909    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg" podStartSLOduration=1.926590668 podStartE2EDuration="6.532890608s" totalImagesPullingTime="4.60629994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:02.532685417 +0000 UTC m=+890.543338728" watchObservedRunningTime="2026-09-07 10:35:02.532890608 +0000 UTC m=+890.543543922"
Sep 07 10:35:08.547514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:08.547482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-22kwg"
Sep 07 10:35:09.369473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.369443    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:35:09.372402 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.372386    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.375792 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.375775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 10:35:09.381216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.381193    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:35:09.411153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.411129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzkg\" (UniqueName: \"kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.411245 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.411166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.512460 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.512438    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzkg\" (UniqueName: \"kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.512537 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.512475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.514827 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.514809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.521650 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.521628    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzkg\" (UniqueName: \"kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg\") pod \"authorino-6fc484f46b-2tjgj\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") " pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.680956 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.680877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:35:09.949056 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:09.948878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:35:09.951645 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:35:09.951620    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf270ba88_a221_40d5_8cf5_1df3d084f929.slice/crio-94253c12ae38f17a1e9c6773a3f94b7d84c551907250ba92e490f9cfa6103afe WatchSource:0}: Error finding container 94253c12ae38f17a1e9c6773a3f94b7d84c551907250ba92e490f9cfa6103afe: Status 404 returned error can't find the container with id 94253c12ae38f17a1e9c6773a3f94b7d84c551907250ba92e490f9cfa6103afe
Sep 07 10:35:10.543408 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:10.543376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" event={"ID":"f270ba88-a221-40d5-8cf5-1df3d084f929","Type":"ContainerStarted","Data":"94253c12ae38f17a1e9c6773a3f94b7d84c551907250ba92e490f9cfa6103afe"}
Sep 07 10:35:11.548649 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.548614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" event={"ID":"f270ba88-a221-40d5-8cf5-1df3d084f929","Type":"ContainerStarted","Data":"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"}
Sep 07 10:35:11.569597 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.569545    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" podStartSLOduration=1.974369695 podStartE2EDuration="2.569529636s" totalImagesPullingTime="595.159941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:11.568643968 +0000 UTC m=+899.579297272" watchObservedRunningTime="2026-09-07 10:35:11.569529636 +0000 UTC m=+899.580182949"
Sep 07 10:35:11.600695 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.600666    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:35:11.600856 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.600836    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" podUID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" containerName="authorino" containerID="cri-o://7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0" gracePeriod=30
Sep 07 10:35:11.889903 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.889878    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:35:11.933518 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.933498    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb2qh\" (UniqueName: \"kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh\") pod \"b77b5a95-ce89-4b97-bcc3-262b6b1f6529\" (UID: \"b77b5a95-ce89-4b97-bcc3-262b6b1f6529\") "
Sep 07 10:35:11.935566 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:11.935537    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh" (OuterVolumeSpecName: "kube-api-access-pb2qh") pod "b77b5a95-ce89-4b97-bcc3-262b6b1f6529" (UID: "b77b5a95-ce89-4b97-bcc3-262b6b1f6529"). InnerVolumeSpecName "kube-api-access-pb2qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:12.034440 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.034415    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2qh\" (UniqueName: \"kubernetes.io/projected/b77b5a95-ce89-4b97-bcc3-262b6b1f6529-kube-api-access-pb2qh\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:35:12.446861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.446833    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:35:12.447035 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.446833    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:35:12.553189 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.553159    2562 generic.go:356] "Generic (PLEG): container finished" podID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" containerID="7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0" exitCode=0
Sep 07 10:35:12.553530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.553200    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-5wbg8"
Sep 07 10:35:12.553530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.553234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" event={"ID":"b77b5a95-ce89-4b97-bcc3-262b6b1f6529","Type":"ContainerDied","Data":"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"}
Sep 07 10:35:12.553530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.553268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-5wbg8" event={"ID":"b77b5a95-ce89-4b97-bcc3-262b6b1f6529","Type":"ContainerDied","Data":"9ab305cdb3f66379c92e495e5b6bc79c379432f3766eea92e69f5dfe9bf2a30b"}
Sep 07 10:35:12.553530 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.553289    2562 scope.go:117] "RemoveContainer" containerID="7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"
Sep 07 10:35:12.560463 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.560420    2562 scope.go:117] "RemoveContainer" containerID="7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"
Sep 07 10:35:12.560663 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:35:12.560647    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0\": container with ID starting with 7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0 not found: ID does not exist" containerID="7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"
Sep 07 10:35:12.560710 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.560670    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0"} err="failed to get container status \"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0\": rpc error: code = NotFound desc = could not find container \"7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0\": container with ID starting with 7644c634276ee76fd6939396bebce0bb5531da15a71d41e9dfb4ccfad5af9bd0 not found: ID does not exist"
Sep 07 10:35:12.571913 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.571890    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:35:12.575935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:12.575917    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-5wbg8"]
Sep 07 10:35:14.494784 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:35:14.494750    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" path="/var/lib/kubelet/pods/b77b5a95-ce89-4b97-bcc3-262b6b1f6529/volumes"
Sep 07 10:36:26.829663 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.829624    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"]
Sep 07 10:36:26.830098 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.830079    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" containerName="authorino"
Sep 07 10:36:26.830145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.830103    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" containerName="authorino"
Sep 07 10:36:26.830189 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.830178    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b77b5a95-ce89-4b97-bcc3-262b6b1f6529" containerName="authorino"
Sep 07 10:36:26.832714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.832696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.835595 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.835575    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v8l9q\""
Sep 07 10:36:26.835712 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.835597    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 10:36:26.836891 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.836868    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 10:36:26.837011 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.836970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 07 10:36:26.848058 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.848037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"]
Sep 07 10:36:26.886652 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.886776 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886658    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e755632e-3f91-43c8-b295-43aa4756622a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.886776 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.886776 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctklm\" (UniqueName: \"kubernetes.io/projected/e755632e-3f91-43c8-b295-43aa4756622a-kube-api-access-ctklm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.886930 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.886930 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.886840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.987990 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.987953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.987990 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.987996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988230 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988230 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e755632e-3f91-43c8-b295-43aa4756622a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988230 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988230 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctklm\" (UniqueName: \"kubernetes.io/projected/e755632e-3f91-43c8-b295-43aa4756622a-kube-api-access-ctklm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988457 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988528 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.988528 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.988492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.990472 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.990450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e755632e-3f91-43c8-b295-43aa4756622a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:26.990658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:26.990642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e755632e-3f91-43c8-b295-43aa4756622a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:27.004432 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:27.004408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctklm\" (UniqueName: \"kubernetes.io/projected/e755632e-3f91-43c8-b295-43aa4756622a-kube-api-access-ctklm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6gwzv\" (UID: \"e755632e-3f91-43c8-b295-43aa4756622a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:27.142805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:27.142722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:27.411897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:27.411820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"]
Sep 07 10:36:27.414719 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:36:27.414696    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode755632e_3f91_43c8_b295_43aa4756622a.slice/crio-f2b83eef0cf98ae1a099aaea4dad89944fbe7811f87cac145e9b52357cb8d4b2 WatchSource:0}: Error finding container f2b83eef0cf98ae1a099aaea4dad89944fbe7811f87cac145e9b52357cb8d4b2: Status 404 returned error can't find the container with id f2b83eef0cf98ae1a099aaea4dad89944fbe7811f87cac145e9b52357cb8d4b2
Sep 07 10:36:27.823950 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:27.823911    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv" event={"ID":"e755632e-3f91-43c8-b295-43aa4756622a","Type":"ContainerStarted","Data":"f2b83eef0cf98ae1a099aaea4dad89944fbe7811f87cac145e9b52357cb8d4b2"}
Sep 07 10:36:28.829581 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:28.829546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv" event={"ID":"e755632e-3f91-43c8-b295-43aa4756622a","Type":"ContainerStarted","Data":"d0cab0ba0671b29820c0329eeb6d61af6388208297eee4307299fad60119b88f"}
Sep 07 10:36:28.849145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:28.849104    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv" podStartSLOduration=1.8026014 podStartE2EDuration="2.84908957s" totalImagesPullingTime="1.04648817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:28.847288711 +0000 UTC m=+976.857942028" watchObservedRunningTime="2026-09-07 10:36:28.84908957 +0000 UTC m=+976.859742926"
Sep 07 10:36:37.143367 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:37.143253    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:37.143367 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:37.143344    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:37.155684 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:37.155657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:37.875206 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:37.875176    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6gwzv"
Sep 07 10:36:41.729703 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.729670    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"]
Sep 07 10:36:41.732805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.732782    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.735511 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.735491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 07 10:36:41.744834 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.744812    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"]
Sep 07 10:36:41.797498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797458    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.797666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.797666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.797666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.797666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.797802 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.797685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmjx\" (UniqueName: \"kubernetes.io/projected/e8740642-e00c-4e33-82ac-4ede3f52e6b3-kube-api-access-7rmjx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898433 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898433 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898430    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmjx\" (UniqueName: \"kubernetes.io/projected/e8740642-e00c-4e33-82ac-4ede3f52e6b3-kube-api-access-7rmjx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898914    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.898944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.899036 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.898967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.900720 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.900703    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8740642-e00c-4e33-82ac-4ede3f52e6b3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.901157 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.901138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8740642-e00c-4e33-82ac-4ede3f52e6b3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:41.907478 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:41.907457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmjx\" (UniqueName: \"kubernetes.io/projected/e8740642-e00c-4e33-82ac-4ede3f52e6b3-kube-api-access-7rmjx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-29v4f\" (UID: \"e8740642-e00c-4e33-82ac-4ede3f52e6b3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:42.042911 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.042822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:42.315470 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.315435    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"]
Sep 07 10:36:42.319317 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:36:42.319277    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8740642_e00c_4e33_82ac_4ede3f52e6b3.slice/crio-5dc8c14677b93750b8e0dd116ef6686c5a041e0bc29229d109e48ecb9146115b WatchSource:0}: Error finding container 5dc8c14677b93750b8e0dd116ef6686c5a041e0bc29229d109e48ecb9146115b: Status 404 returned error can't find the container with id 5dc8c14677b93750b8e0dd116ef6686c5a041e0bc29229d109e48ecb9146115b
Sep 07 10:36:42.320921 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.320899    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:36:42.882242 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.882154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f" event={"ID":"e8740642-e00c-4e33-82ac-4ede3f52e6b3","Type":"ContainerStarted","Data":"da5c01bed5b3a64256dce2cc0f7d2951095323974afc2e25f57873d91cc395e3"}
Sep 07 10:36:42.882242 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.882221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f" event={"ID":"e8740642-e00c-4e33-82ac-4ede3f52e6b3","Type":"ContainerStarted","Data":"5dc8c14677b93750b8e0dd116ef6686c5a041e0bc29229d109e48ecb9146115b"}
Sep 07 10:36:42.905261 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:42.904447    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f" podStartSLOduration=1.6907622359999999 podStartE2EDuration="1.904424878s" totalImagesPullingTime="213.662642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:42.900326983 +0000 UTC m=+990.910980289" watchObservedRunningTime="2026-09-07 10:36:42.904424878 +0000 UTC m=+990.915078194"
Sep 07 10:36:52.043152 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:52.043096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:52.043643 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:52.043175    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:52.055968 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:52.055946    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:36:52.932979 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:36:52.932943    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-29v4f"
Sep 07 10:37:08.331592 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.331564    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"]
Sep 07 10:37:08.334839 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.334821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.337819 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.337789    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 07 10:37:08.346803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.346784    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"]
Sep 07 10:37:08.418868 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.418833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr65l\" (UniqueName: \"kubernetes.io/projected/e3aed68f-f782-43f2-a816-5bfd4c498cd7-kube-api-access-tr65l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.419027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.418880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.419027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.418913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.419027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.418964    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.419027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.418991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.419027 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.419008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519361 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519483 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519483 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519580 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519636 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr65l\" (UniqueName: \"kubernetes.io/projected/e3aed68f-f782-43f2-a816-5bfd4c498cd7-kube-api-access-tr65l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519684 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519778 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.519865 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.520082 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.519870    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.521847 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.521825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3aed68f-f782-43f2-a816-5bfd4c498cd7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.522263 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.522243    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3aed68f-f782-43f2-a816-5bfd4c498cd7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.528469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.528446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr65l\" (UniqueName: \"kubernetes.io/projected/e3aed68f-f782-43f2-a816-5bfd4c498cd7-kube-api-access-tr65l\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-phn72\" (UID: \"e3aed68f-f782-43f2-a816-5bfd4c498cd7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.646340 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.646223    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:08.918435 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.918388    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"]
Sep 07 10:37:08.921091 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:37:08.921061    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3aed68f_f782_43f2_a816_5bfd4c498cd7.slice/crio-88ee4cd53f0b55fc1d1d4da798433d0f69e28cd94758bef3b0d58587728ce2fe WatchSource:0}: Error finding container 88ee4cd53f0b55fc1d1d4da798433d0f69e28cd94758bef3b0d58587728ce2fe: Status 404 returned error can't find the container with id 88ee4cd53f0b55fc1d1d4da798433d0f69e28cd94758bef3b0d58587728ce2fe
Sep 07 10:37:08.983591 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:08.983555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72" event={"ID":"e3aed68f-f782-43f2-a816-5bfd4c498cd7","Type":"ContainerStarted","Data":"88ee4cd53f0b55fc1d1d4da798433d0f69e28cd94758bef3b0d58587728ce2fe"}
Sep 07 10:37:09.989129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:09.989091    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72" event={"ID":"e3aed68f-f782-43f2-a816-5bfd4c498cd7","Type":"ContainerStarted","Data":"c689c310b9be4a51f32c963607ace4b4b774489297bab5b9e969d541f3ee875d"}
Sep 07 10:37:10.011083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:10.011030    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72" podStartSLOduration=1.822758672 podStartE2EDuration="2.011010162s" totalImagesPullingTime="188.25149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:10.009648207 +0000 UTC m=+1018.020301524" watchObservedRunningTime="2026-09-07 10:37:10.011010162 +0000 UTC m=+1018.021663478"
Sep 07 10:37:17.832932 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.832893    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"]
Sep 07 10:37:17.836590 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.836569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.839453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.839429    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 10:37:17.845031 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.845007    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"]
Sep 07 10:37:17.999081 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999037    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.999081 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.999379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tw4l\" (UniqueName: \"kubernetes.io/projected/820292cb-5028-44e9-ab4b-ed5fe71e7fba-kube-api-access-6tw4l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.999379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.999379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:17.999379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:17.999362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100515 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100515 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100485    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tw4l\" (UniqueName: \"kubernetes.io/projected/820292cb-5028-44e9-ab4b-ed5fe71e7fba-kube-api-access-6tw4l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100515 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100780 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100780 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100780 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.100957 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.101037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.100958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.101037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.101016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.102972 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.102944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/820292cb-5028-44e9-ab4b-ed5fe71e7fba-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.103195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.103180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/820292cb-5028-44e9-ab4b-ed5fe71e7fba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.109843 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.109820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tw4l\" (UniqueName: \"kubernetes.io/projected/820292cb-5028-44e9-ab4b-ed5fe71e7fba-kube-api-access-6tw4l\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8\" (UID: \"820292cb-5028-44e9-ab4b-ed5fe71e7fba\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.145858 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.145808    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:18.420977 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.420949    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"]
Sep 07 10:37:18.423386 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:37:18.423355    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod820292cb_5028_44e9_ab4b_ed5fe71e7fba.slice/crio-b50290d2a2ceab85c19dc3f67cced6e59d174b50740795680f43aede067564b1 WatchSource:0}: Error finding container b50290d2a2ceab85c19dc3f67cced6e59d174b50740795680f43aede067564b1: Status 404 returned error can't find the container with id b50290d2a2ceab85c19dc3f67cced6e59d174b50740795680f43aede067564b1
Sep 07 10:37:18.646625 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.646590    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:18.646749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.646677    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:18.660040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:18.660008    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:19.032326 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:19.032263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8" event={"ID":"820292cb-5028-44e9-ab4b-ed5fe71e7fba","Type":"ContainerStarted","Data":"de7a4ddf2aeff8489432afc8f73a241a37e9ad7a1fa1b86ab5256fad427a87a4"}
Sep 07 10:37:19.032326 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:19.032325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8" event={"ID":"820292cb-5028-44e9-ab4b-ed5fe71e7fba","Type":"ContainerStarted","Data":"b50290d2a2ceab85c19dc3f67cced6e59d174b50740795680f43aede067564b1"}
Sep 07 10:37:19.043948 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:19.043917    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-phn72"
Sep 07 10:37:19.051587 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:19.051531    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8" podStartSLOduration=1.8880566829999998 podStartE2EDuration="2.051512353s" totalImagesPullingTime="163.45567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:19.050771254 +0000 UTC m=+1027.061424570" watchObservedRunningTime="2026-09-07 10:37:19.051512353 +0000 UTC m=+1027.062165668"
Sep 07 10:37:28.146096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:28.146054    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:28.146570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:28.146111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:28.158545 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:28.158521    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:37:29.082805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:37:29.082777    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8"
Sep 07 10:38:07.567477 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.567444    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56d56b9c8c-txqjs"]
Sep 07 10:38:07.570740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.570722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.579153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.579129    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56d56b9c8c-txqjs"]
Sep 07 10:38:07.684145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.684117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-tls-cert\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.684349 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.684161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7k9\" (UniqueName: \"kubernetes.io/projected/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-kube-api-access-mq7k9\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.784838 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.784788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7k9\" (UniqueName: \"kubernetes.io/projected/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-kube-api-access-mq7k9\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.785000 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.784890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-tls-cert\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.787553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.787523    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-tls-cert\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.796734 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.796711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7k9\" (UniqueName: \"kubernetes.io/projected/3c5f1dbd-4ffc-45f3-8522-906a0ef963d9-kube-api-access-mq7k9\") pod \"authorino-56d56b9c8c-txqjs\" (UID: \"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9\") " pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:07.880165 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:07.880094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56d56b9c8c-txqjs"
Sep 07 10:38:08.148035 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:08.147954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56d56b9c8c-txqjs"]
Sep 07 10:38:08.150962 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:38:08.150930    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c5f1dbd_4ffc_45f3_8522_906a0ef963d9.slice/crio-dd65f54c6735a517c217e0c1cb7286d037c683f6903bbcafc61349fee20019ad WatchSource:0}: Error finding container dd65f54c6735a517c217e0c1cb7286d037c683f6903bbcafc61349fee20019ad: Status 404 returned error can't find the container with id dd65f54c6735a517c217e0c1cb7286d037c683f6903bbcafc61349fee20019ad
Sep 07 10:38:08.233166 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:08.233133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d56b9c8c-txqjs" event={"ID":"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9","Type":"ContainerStarted","Data":"dd65f54c6735a517c217e0c1cb7286d037c683f6903bbcafc61349fee20019ad"}
Sep 07 10:38:09.240514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.240412    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56d56b9c8c-txqjs" event={"ID":"3c5f1dbd-4ffc-45f3-8522-906a0ef963d9","Type":"ContainerStarted","Data":"6f837cf245137b87f656466183c1ccc0c57794292adddcd526e777d1baba18d9"}
Sep 07 10:38:09.261847 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.261780    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56d56b9c8c-txqjs" podStartSLOduration=1.539549627 podStartE2EDuration="2.261758013s" totalImagesPullingTime="722.208386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:09.260981314 +0000 UTC m=+1077.271634628" watchObservedRunningTime="2026-09-07 10:38:09.261758013 +0000 UTC m=+1077.272411329"
Sep 07 10:38:09.295330 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.293483    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:38:09.304632 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.293774    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" podUID="f270ba88-a221-40d5-8cf5-1df3d084f929" containerName="authorino" containerID="cri-o://0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7" gracePeriod=30
Sep 07 10:38:09.563629 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.563597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:38:09.608094 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.608057    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjzkg\" (UniqueName: \"kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg\") pod \"f270ba88-a221-40d5-8cf5-1df3d084f929\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") "
Sep 07 10:38:09.608094 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.608101    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert\") pod \"f270ba88-a221-40d5-8cf5-1df3d084f929\" (UID: \"f270ba88-a221-40d5-8cf5-1df3d084f929\") "
Sep 07 10:38:09.610628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.610593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg" (OuterVolumeSpecName: "kube-api-access-pjzkg") pod "f270ba88-a221-40d5-8cf5-1df3d084f929" (UID: "f270ba88-a221-40d5-8cf5-1df3d084f929"). InnerVolumeSpecName "kube-api-access-pjzkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:38:09.618592 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.618563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "f270ba88-a221-40d5-8cf5-1df3d084f929" (UID: "f270ba88-a221-40d5-8cf5-1df3d084f929"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:38:09.709176 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.709143    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzkg\" (UniqueName: \"kubernetes.io/projected/f270ba88-a221-40d5-8cf5-1df3d084f929-kube-api-access-pjzkg\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:38:09.709176 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:09.709170    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f270ba88-a221-40d5-8cf5-1df3d084f929-tls-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:38:10.245258 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.245181    2562 generic.go:356] "Generic (PLEG): container finished" podID="f270ba88-a221-40d5-8cf5-1df3d084f929" containerID="0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7" exitCode=0
Sep 07 10:38:10.245258 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.245222    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fc484f46b-2tjgj"
Sep 07 10:38:10.245793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.245279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" event={"ID":"f270ba88-a221-40d5-8cf5-1df3d084f929","Type":"ContainerDied","Data":"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"}
Sep 07 10:38:10.245793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.245354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fc484f46b-2tjgj" event={"ID":"f270ba88-a221-40d5-8cf5-1df3d084f929","Type":"ContainerDied","Data":"94253c12ae38f17a1e9c6773a3f94b7d84c551907250ba92e490f9cfa6103afe"}
Sep 07 10:38:10.245793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.245401    2562 scope.go:117] "RemoveContainer" containerID="0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"
Sep 07 10:38:10.254046 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.254028    2562 scope.go:117] "RemoveContainer" containerID="0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"
Sep 07 10:38:10.254316 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:38:10.254284    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7\": container with ID starting with 0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7 not found: ID does not exist" containerID="0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"
Sep 07 10:38:10.254369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.254328    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7"} err="failed to get container status \"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7\": rpc error: code = NotFound desc = could not find container \"0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7\": container with ID starting with 0ad333ad87702d362d2cd349a705a7011458c89987ea9e0c8baff9029df491b7 not found: ID does not exist"
Sep 07 10:38:10.266534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.266490    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:38:10.270812 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.270786    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fc484f46b-2tjgj"]
Sep 07 10:38:10.496294 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:10.496219    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f270ba88-a221-40d5-8cf5-1df3d084f929" path="/var/lib/kubelet/pods/f270ba88-a221-40d5-8cf5-1df3d084f929/volumes"
Sep 07 10:38:41.112025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.111995    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:38:41.112412 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.112277    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f270ba88-a221-40d5-8cf5-1df3d084f929" containerName="authorino"
Sep 07 10:38:41.112412 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.112287    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f270ba88-a221-40d5-8cf5-1df3d084f929" containerName="authorino"
Sep 07 10:38:41.112412 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.112371    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f270ba88-a221-40d5-8cf5-1df3d084f929" containerName="authorino"
Sep 07 10:38:41.115011 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.114997    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.117679 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.117659    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea745819\""
Sep 07 10:38:41.117812 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.117746    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea745819-dockercfg-lsxfl\""
Sep 07 10:38:41.133210 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.133188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:38:41.265002 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.264976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxc5\" (UniqueName: \"kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.265164 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.265021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.366216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.366155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.366419 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.366218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxc5\" (UniqueName: \"kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.366828 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.366811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.374441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.374413    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxc5\" (UniqueName: \"kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5\") pod \"payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.425015 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.424993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:41.694139 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:41.694064    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:38:41.696897 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:38:41.696865    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8295b0e4_9331_44e3_9997_716baadc9afe.slice/crio-c3b69a918c87c1be556f587b3656c42e04017f69102496bbc4e59e58d130c842 WatchSource:0}: Error finding container c3b69a918c87c1be556f587b3656c42e04017f69102496bbc4e59e58d130c842: Status 404 returned error can't find the container with id c3b69a918c87c1be556f587b3656c42e04017f69102496bbc4e59e58d130c842
Sep 07 10:38:42.380529 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:42.380490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" event={"ID":"8295b0e4-9331-44e3-9997-716baadc9afe","Type":"ContainerStarted","Data":"c3b69a918c87c1be556f587b3656c42e04017f69102496bbc4e59e58d130c842"}
Sep 07 10:38:43.772704 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.772675    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:38:43.776266 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.776245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.778814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.778748    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c5b29a\""
Sep 07 10:38:43.778814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.778749    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c5b29a-dockercfg-swj4g\""
Sep 07 10:38:43.786164 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.786142    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:38:43.887658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.887625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.887824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.887683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58ql\" (UniqueName: \"kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.989089 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.989009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58ql\" (UniqueName: \"kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.989172 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.989124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.990038 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.990016    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:43.998693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:43.998663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58ql\" (UniqueName: \"kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql\") pod \"payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") " pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:44.089393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.089348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:44.371186 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.371156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:38:44.374456 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:38:44.374435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e9e650_573e_41c9_a155_52a0720a1ce9.slice/crio-0f310a379187af7625b228cad74cf5358137b7937995d47196c5a77abf16d5fe WatchSource:0}: Error finding container 0f310a379187af7625b228cad74cf5358137b7937995d47196c5a77abf16d5fe: Status 404 returned error can't find the container with id 0f310a379187af7625b228cad74cf5358137b7937995d47196c5a77abf16d5fe
Sep 07 10:38:44.390658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.390628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" event={"ID":"8295b0e4-9331-44e3-9997-716baadc9afe","Type":"ContainerStarted","Data":"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"}
Sep 07 10:38:44.390775 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.390723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:44.392003 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.391981    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" event={"ID":"07e9e650-573e-41c9-a155-52a0720a1ce9","Type":"ContainerStarted","Data":"0f310a379187af7625b228cad74cf5358137b7937995d47196c5a77abf16d5fe"}
Sep 07 10:38:44.409378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:44.409266    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" podStartSLOduration=1.131953901 podStartE2EDuration="3.409247648s" totalImagesPullingTime="2.277293747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:44.409106854 +0000 UTC m=+1112.419760167" watchObservedRunningTime="2026-09-07 10:38:44.409247648 +0000 UTC m=+1112.419900961"
Sep 07 10:38:45.398336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:45.398287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" event={"ID":"07e9e650-573e-41c9-a155-52a0720a1ce9","Type":"ContainerStarted","Data":"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"}
Sep 07 10:38:45.398719 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:45.398482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:45.414228 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:45.414181    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" podStartSLOduration=2.414163925 podStartE2EDuration="2.414163925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:45.413451618 +0000 UTC m=+1113.424104957" watchObservedRunningTime="2026-09-07 10:38:45.414163925 +0000 UTC m=+1113.424817241"
Sep 07 10:38:55.400712 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:55.400682    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:38:56.404670 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:56.404643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:38:57.639365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.639331    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:38:57.644064 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.644044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.646713 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.646692    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 10:38:57.646846 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.646730    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b4061f-openshift-default-dockercfg-rcgz8\""
Sep 07 10:38:57.656098 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.656077    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:38:57.693342 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693458 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693572 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693572 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693665 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693665 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.693665 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.693655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb92v\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794035 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794092    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb92v\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794536 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794423    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.794908 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.794898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.796691 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.796670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.796897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.796877    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.801993 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.801965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.802228 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.802209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb92v\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v\") pod \"e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") " pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:57.954923 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:57.954852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:38:58.231565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:58.231542    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:38:58.234174 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:38:58.234148    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd7ec01_4ad1_4c12_8482_014617e5fb6e.slice/crio-a8b0fcfe7e98071da5db2fdaefebf88191602291d2a44648837bc62fe0434251 WatchSource:0}: Error finding container a8b0fcfe7e98071da5db2fdaefebf88191602291d2a44648837bc62fe0434251: Status 404 returned error can't find the container with id a8b0fcfe7e98071da5db2fdaefebf88191602291d2a44648837bc62fe0434251
Sep 07 10:38:58.456142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:38:58.456114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerStarted","Data":"a8b0fcfe7e98071da5db2fdaefebf88191602291d2a44648837bc62fe0434251"}
Sep 07 10:39:00.860612 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:00.860573    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:00.860953 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:00.860659    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:00.860953 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:00.860706    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:01.470583 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:01.470545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerStarted","Data":"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"}
Sep 07 10:39:01.492028 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:01.491966    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podStartSLOduration=1.868416753 podStartE2EDuration="4.49194655s" totalImagesPullingTime="2.623529797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:01.491334482 +0000 UTC m=+1129.501987797" watchObservedRunningTime="2026-09-07 10:39:01.49194655 +0000 UTC m=+1129.502599864"
Sep 07 10:39:01.955548 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:01.955516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:01.960188 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:01.960167    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:02.474956 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:02.474930    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:02.475812 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:02.475796    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:04.989262 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:04.989229    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"]
Sep 07 10:39:04.992657 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:04.992642    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:04.995320 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:04.995278    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:39:05.002678 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.002652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"]
Sep 07 10:39:05.051021 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.050982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea745819-nbv8v\" (UID: \"8ee5b197-67f7-4675-9520-e6ef453494ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:05.152318 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.152257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea745819-nbv8v\" (UID: \"8ee5b197-67f7-4675-9520-e6ef453494ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:05.152856 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.152837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ea745819-nbv8v\" (UID: \"8ee5b197-67f7-4675-9520-e6ef453494ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:05.302631 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.302570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:05.583413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.583388    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"]
Sep 07 10:39:05.586438 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:39:05.586409    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee5b197_67f7_4675_9520_e6ef453494ba.slice/crio-507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf WatchSource:0}: Error finding container 507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf: Status 404 returned error can't find the container with id 507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf
Sep 07 10:39:05.620231 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.620204    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:39:05.624600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.624580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.627132 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.627110    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b4061f-dockercfg-cn9kl\""
Sep 07 10:39:05.627340 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.627327    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b4061f\""
Sep 07 10:39:05.634556 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.634535    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:39:05.656986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.656962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bjg\" (UniqueName: \"kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.657075 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.657010    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.757932 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.757903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.758085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.757981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bjg\" (UniqueName: \"kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.758602 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.758583    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.767050 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.767019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bjg\" (UniqueName: \"kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg\") pod \"payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:05.936378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:05.936278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:06.219204 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.219133    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:39:06.222815 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:39:06.222786    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dfaa093_66b5_4ee4_8029_ecd56cf4d314.slice/crio-dfb6f215fc0ac5d364170a32862a104b0e37af627b3074d2e8bb2cf5f9c8bf03 WatchSource:0}: Error finding container dfb6f215fc0ac5d364170a32862a104b0e37af627b3074d2e8bb2cf5f9c8bf03: Status 404 returned error can't find the container with id dfb6f215fc0ac5d364170a32862a104b0e37af627b3074d2e8bb2cf5f9c8bf03
Sep 07 10:39:06.496964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.496822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" event={"ID":"1dfaa093-66b5-4ee4-8029-ecd56cf4d314","Type":"ContainerStarted","Data":"afa81fa7c0215c3b478dafca0349eaacd5f7289412ddc2ad94f0ea10af320dec"}
Sep 07 10:39:06.496964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.496876    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" event={"ID":"1dfaa093-66b5-4ee4-8029-ecd56cf4d314","Type":"ContainerStarted","Data":"dfb6f215fc0ac5d364170a32862a104b0e37af627b3074d2e8bb2cf5f9c8bf03"}
Sep 07 10:39:06.496964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.496904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:06.497654 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.497592    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v" event={"ID":"8ee5b197-67f7-4675-9520-e6ef453494ba","Type":"ContainerStarted","Data":"507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf"}
Sep 07 10:39:06.520990 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:06.520923    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" podStartSLOduration=1.520903415 podStartE2EDuration="1.520903415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:06.519437379 +0000 UTC m=+1134.530090696" watchObservedRunningTime="2026-09-07 10:39:06.520903415 +0000 UTC m=+1134.531556729"
Sep 07 10:39:08.508649 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:08.508609    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ee5b197-67f7-4675-9520-e6ef453494ba" containerID="9d74ddeb2dbce4c7909bdb59650493b446e04be3c69d34b2df63900a1e62f5a4" exitCode=0
Sep 07 10:39:08.508989 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:08.508697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v" event={"ID":"8ee5b197-67f7-4675-9520-e6ef453494ba","Type":"ContainerDied","Data":"9d74ddeb2dbce4c7909bdb59650493b446e04be3c69d34b2df63900a1e62f5a4"}
Sep 07 10:39:09.640163 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:09.640141    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:09.697410 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:09.697357    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca\") pod \"8ee5b197-67f7-4675-9520-e6ef453494ba\" (UID: \"8ee5b197-67f7-4675-9520-e6ef453494ba\") "
Sep 07 10:39:09.697674 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:09.697651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ee5b197-67f7-4675-9520-e6ef453494ba" (UID: "8ee5b197-67f7-4675-9520-e6ef453494ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:39:09.798281 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:09.798198    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5b197-67f7-4675-9520-e6ef453494ba-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:39:10.518041 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:10.518011    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"
Sep 07 10:39:10.518041 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:10.518029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v" event={"ID":"8ee5b197-67f7-4675-9520-e6ef453494ba","Type":"ContainerDied","Data":"507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf"}
Sep 07 10:39:10.518200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:10.518072    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="507df6144f15c8810ff614c32870e9b2ff6f44708058a9b6fc133a47f9ebd9bf"
Sep 07 10:39:17.321354 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.321295    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:39:17.321835 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.321813    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee5b197-67f7-4675-9520-e6ef453494ba" containerName="revoke-keys"
Sep 07 10:39:17.321876 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.321840    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee5b197-67f7-4675-9520-e6ef453494ba" containerName="revoke-keys"
Sep 07 10:39:17.321961 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.321948    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee5b197-67f7-4675-9520-e6ef453494ba" containerName="revoke-keys"
Sep 07 10:39:17.330045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.330026    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.333386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.333343    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-36798ca4-kserve-self-signed-certs\""
Sep 07 10:39:17.336104 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.336086    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:39:17.372285 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372251    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.372424 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.372424 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.372509 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.372549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372517    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.372549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.372539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfvt\" (UniqueName: \"kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473644    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfvt\" (UniqueName: \"kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473898 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.473898 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.473776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.474092 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.474064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.474167 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.474104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.474212 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.474165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.476068 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.476046    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.476290 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.476267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.482721 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.482697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfvt\" (UniqueName: \"kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt\") pod \"e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") " pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.505362 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.505323    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:39:17.640564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.640476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:17.915145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:17.915073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:39:17.918025 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:39:17.918001    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfe0d624_2bcb_4ac6_866c_21f6748ce557.slice/crio-3b512822866c55c0c26101ff04c10f863e464fa810a28d0a9dfd2d56730a0074 WatchSource:0}: Error finding container 3b512822866c55c0c26101ff04c10f863e464fa810a28d0a9dfd2d56730a0074: Status 404 returned error can't find the container with id 3b512822866c55c0c26101ff04c10f863e464fa810a28d0a9dfd2d56730a0074
Sep 07 10:39:18.516461 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:18.516429    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 10:39:18.516821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:18.516476    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 10:39:18.568560 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:18.568527    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" event={"ID":"dfe0d624-2bcb-4ac6-866c-21f6748ce557","Type":"ContainerStarted","Data":"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"}
Sep 07 10:39:18.568714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:18.568568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" event={"ID":"dfe0d624-2bcb-4ac6-866c-21f6748ce557","Type":"ContainerStarted","Data":"3b512822866c55c0c26101ff04c10f863e464fa810a28d0a9dfd2d56730a0074"}
Sep 07 10:39:18.586289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:18.586246    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" podStartSLOduration=1.586232716 podStartE2EDuration="1.586232716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:18.584464729 +0000 UTC m=+1146.595118045" watchObservedRunningTime="2026-09-07 10:39:18.586232716 +0000 UTC m=+1146.596886050"
Sep 07 10:39:19.574586 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.574558    2562 generic.go:356] "Generic (PLEG): container finished" podID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerID="3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296" exitCode=0
Sep 07 10:39:19.575076 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.574643    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerDied","Data":"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"}
Sep 07 10:39:19.575152 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.575108    2562 scope.go:117] "RemoveContainer" containerID="3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"
Sep 07 10:39:19.576189 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.576161    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:19.576246 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.576230    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:19.576279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:19.576261    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:39:20.580533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:20.580502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerStarted","Data":"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"}
Sep 07 10:39:20.955508 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:20.955415    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:20.955682 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:20.955631    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 07 10:39:20.955751 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:20.955683    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 07 10:39:21.245581 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.245497    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:39:21.245738 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.245697    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" podUID="8295b0e4-9331-44e3-9997-716baadc9afe" containerName="payload-pre-processing" containerID="cri-o://a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25" gracePeriod=30
Sep 07 10:39:21.413340 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.412372    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"]
Sep 07 10:39:21.417322 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.415060    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ea745819-nbv8v"]
Sep 07 10:39:21.532451 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.532345    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:39:21.586208 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.586171    2562 generic.go:356] "Generic (PLEG): container finished" podID="8295b0e4-9331-44e3-9997-716baadc9afe" containerID="a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25" exitCode=0
Sep 07 10:39:21.586648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.586219    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"
Sep 07 10:39:21.586648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.586263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" event={"ID":"8295b0e4-9331-44e3-9997-716baadc9afe","Type":"ContainerDied","Data":"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"}
Sep 07 10:39:21.586648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.586343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr" event={"ID":"8295b0e4-9331-44e3-9997-716baadc9afe","Type":"ContainerDied","Data":"c3b69a918c87c1be556f587b3656c42e04017f69102496bbc4e59e58d130c842"}
Sep 07 10:39:21.586648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.586379    2562 scope.go:117] "RemoveContainer" containerID="a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"
Sep 07 10:39:21.594468 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.594451    2562 scope.go:117] "RemoveContainer" containerID="a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"
Sep 07 10:39:21.594727 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:39:21.594710    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25\": container with ID starting with a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25 not found: ID does not exist" containerID="a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"
Sep 07 10:39:21.594776 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.594736    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25"} err="failed to get container status \"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25\": rpc error: code = NotFound desc = could not find container \"a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25\": container with ID starting with a0b0c2d1dcc7f8ce0a4a84a8c2ae3454e4ca81fd788c701e3d5c4c5ce98d1e25 not found: ID does not exist"
Sep 07 10:39:21.623037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.623011    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume\") pod \"8295b0e4-9331-44e3-9997-716baadc9afe\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") "
Sep 07 10:39:21.623188 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.623059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmxc5\" (UniqueName: \"kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5\") pod \"8295b0e4-9331-44e3-9997-716baadc9afe\" (UID: \"8295b0e4-9331-44e3-9997-716baadc9afe\") "
Sep 07 10:39:21.623373 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.623350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8295b0e4-9331-44e3-9997-716baadc9afe" (UID: "8295b0e4-9331-44e3-9997-716baadc9afe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:39:21.625426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.625404    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5" (OuterVolumeSpecName: "kube-api-access-fmxc5") pod "8295b0e4-9331-44e3-9997-716baadc9afe" (UID: "8295b0e4-9331-44e3-9997-716baadc9afe"). InnerVolumeSpecName "kube-api-access-fmxc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:39:21.724693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.724656    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8295b0e4-9331-44e3-9997-716baadc9afe-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:39:21.724693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.724686    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxc5\" (UniqueName: \"kubernetes.io/projected/8295b0e4-9331-44e3-9997-716baadc9afe-kube-api-access-fmxc5\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:39:21.907993 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.907956    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:39:21.912570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.912523    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ea745819-5458dfbcd6-f2slr"]
Sep 07 10:39:21.956218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.956192    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 07 10:39:21.956346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:21.956236    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 07 10:39:22.495473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:22.495443    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8295b0e4-9331-44e3-9997-716baadc9afe" path="/var/lib/kubelet/pods/8295b0e4-9331-44e3-9997-716baadc9afe/volumes"
Sep 07 10:39:22.495762 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:22.495750    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee5b197-67f7-4675-9520-e6ef453494ba" path="/var/lib/kubelet/pods/8ee5b197-67f7-4675-9520-e6ef453494ba/volumes"
Sep 07 10:39:22.955459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:22.955423    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 07 10:39:22.955841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:22.955479    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 07 10:39:23.955694 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:23.955657    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 07 10:39:23.956065 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:23.955722    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 07 10:39:24.955689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:24.955657    2562 patch_prober.go:28] interesting pod/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 07 10:39:24.955850 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:24.955722    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 07 10:39:25.959901 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:25.959873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:25.960266 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:25.960245    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:25.960894 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:25.960872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:39:27.326893 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.326860    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:39:27.327369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.327334    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8295b0e4-9331-44e3-9997-716baadc9afe" containerName="payload-pre-processing"
Sep 07 10:39:27.327369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.327353    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8295b0e4-9331-44e3-9997-716baadc9afe" containerName="payload-pre-processing"
Sep 07 10:39:27.327497 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.327430    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8295b0e4-9331-44e3-9997-716baadc9afe" containerName="payload-pre-processing"
Sep 07 10:39:27.331751 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.331732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.334354 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.334334    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6cd248c6\""
Sep 07 10:39:27.334462 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.334445    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6cd248c6-dockercfg-wbsrz\""
Sep 07 10:39:27.341375 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.341352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:39:27.472887 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.472858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.473053 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.472903    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrsvq\" (UniqueName: \"kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.574239 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.574208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrsvq\" (UniqueName: \"kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.574450 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.574319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.574994 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.574973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.591772 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.591702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrsvq\" (UniqueName: \"kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq\") pod \"payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") " pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.641234 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.641210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:27.641564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.641546    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:27.641845 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.641830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:27.655443 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.655425    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:27.915343 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:27.915244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:39:27.919925 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:39:27.919899    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42344fac_7107_4151_b6c2_c4d196bc53e4.slice/crio-63affa16109af3793a0253f88f507a78d71cf0f7fa9948a91c7c391010fbd868 WatchSource:0}: Error finding container 63affa16109af3793a0253f88f507a78d71cf0f7fa9948a91c7c391010fbd868: Status 404 returned error can't find the container with id 63affa16109af3793a0253f88f507a78d71cf0f7fa9948a91c7c391010fbd868
Sep 07 10:39:28.621626 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:28.621588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" event={"ID":"42344fac-7107-4151-b6c2-c4d196bc53e4","Type":"ContainerStarted","Data":"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"}
Sep 07 10:39:28.622086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:28.621646    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" event={"ID":"42344fac-7107-4151-b6c2-c4d196bc53e4","Type":"ContainerStarted","Data":"63affa16109af3793a0253f88f507a78d71cf0f7fa9948a91c7c391010fbd868"}
Sep 07 10:39:28.622086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:28.621675    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:28.633392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:28.633366    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:39:28.639945 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:28.639905    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" podStartSLOduration=1.639891661 podStartE2EDuration="1.639891661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:28.63848534 +0000 UTC m=+1156.649138653" watchObservedRunningTime="2026-09-07 10:39:28.639891661 +0000 UTC m=+1156.650545025"
Sep 07 10:39:39.628842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:39.628766    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:39:47.125225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.125182    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:39:47.131784 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.131764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.135574 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.135554    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c5b29a\"/\"kube-root-ca.crt\""
Sep 07 10:39:47.136901 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.136882    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c5b29a\"/\"autores-model-c5b29a-kserve-self-signed-certs\""
Sep 07 10:39:47.136997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.136904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c5b29a\"/\"default-dockercfg-85ltz\""
Sep 07 10:39:47.136997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.136932    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c5b29a\"/\"openshift-service-ca.crt\""
Sep 07 10:39:47.144732 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.144705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:39:47.228045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228022    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.228191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.228191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228070    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.228324 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7nr\" (UniqueName: \"kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.228324 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.228429 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.228331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329370 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329395    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7nr\" (UniqueName: \"kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.329906 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.329836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.331795 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.331770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.332145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.332124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.338166 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.338143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7nr\" (UniqueName: \"kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr\") pod \"autores-model-c5b29a-kserve-76c7c6ff99-5mx8h\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") " pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.440328 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.440261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:47.709464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:47.709433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:39:47.711409 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:39:47.711385    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda34e0f14_f561_4fc1_a8c3_b52a22df764b.slice/crio-280fae58a5b74abbe2c0163935daecee7660bd6c23abf07ca51fd5b32b078660 WatchSource:0}: Error finding container 280fae58a5b74abbe2c0163935daecee7660bd6c23abf07ca51fd5b32b078660: Status 404 returned error can't find the container with id 280fae58a5b74abbe2c0163935daecee7660bd6c23abf07ca51fd5b32b078660
Sep 07 10:39:48.712351 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:48.712293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" event={"ID":"a34e0f14-f561-4fc1-a8c3-b52a22df764b","Type":"ContainerStarted","Data":"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"}
Sep 07 10:39:48.712737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:48.712359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" event={"ID":"a34e0f14-f561-4fc1-a8c3-b52a22df764b","Type":"ContainerStarted","Data":"280fae58a5b74abbe2c0163935daecee7660bd6c23abf07ca51fd5b32b078660"}
Sep 07 10:39:48.730180 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:48.730132    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" podStartSLOduration=1.730118433 podStartE2EDuration="1.730118433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:48.727637703 +0000 UTC m=+1176.738291016" watchObservedRunningTime="2026-09-07 10:39:48.730118433 +0000 UTC m=+1176.740771779"
Sep 07 10:39:57.440891 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:57.440854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:57.440891 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:57.440900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:57.452937 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:57.452911    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:39:57.761725 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:39:57.761660    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:40:06.945235 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:06.945201    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:40:06.945609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:06.945463    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" podUID="42344fac-7107-4151-b6c2-c4d196bc53e4" containerName="payload-processing" containerID="cri-o://0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d" gracePeriod=30
Sep 07 10:40:07.190483 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.190458    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:40:07.295722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.295637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrsvq\" (UniqueName: \"kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq\") pod \"42344fac-7107-4151-b6c2-c4d196bc53e4\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") "
Sep 07 10:40:07.295722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.295676    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume\") pod \"42344fac-7107-4151-b6c2-c4d196bc53e4\" (UID: \"42344fac-7107-4151-b6c2-c4d196bc53e4\") "
Sep 07 10:40:07.296071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.296045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42344fac-7107-4151-b6c2-c4d196bc53e4" (UID: "42344fac-7107-4151-b6c2-c4d196bc53e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:07.297805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.297781    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq" (OuterVolumeSpecName: "kube-api-access-jrsvq") pod "42344fac-7107-4151-b6c2-c4d196bc53e4" (UID: "42344fac-7107-4151-b6c2-c4d196bc53e4"). InnerVolumeSpecName "kube-api-access-jrsvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:07.397279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.397248    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrsvq\" (UniqueName: \"kubernetes.io/projected/42344fac-7107-4151-b6c2-c4d196bc53e4-kube-api-access-jrsvq\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.397279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.397276    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42344fac-7107-4151-b6c2-c4d196bc53e4-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.797485 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.797450    2562 generic.go:356] "Generic (PLEG): container finished" podID="42344fac-7107-4151-b6c2-c4d196bc53e4" containerID="0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d" exitCode=0
Sep 07 10:40:07.797647 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.797495    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"
Sep 07 10:40:07.797647 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.797533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" event={"ID":"42344fac-7107-4151-b6c2-c4d196bc53e4","Type":"ContainerDied","Data":"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"}
Sep 07 10:40:07.797647 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.797584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k" event={"ID":"42344fac-7107-4151-b6c2-c4d196bc53e4","Type":"ContainerDied","Data":"63affa16109af3793a0253f88f507a78d71cf0f7fa9948a91c7c391010fbd868"}
Sep 07 10:40:07.797647 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.797606    2562 scope.go:117] "RemoveContainer" containerID="0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"
Sep 07 10:40:07.808213 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.808192    2562 scope.go:117] "RemoveContainer" containerID="0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"
Sep 07 10:40:07.808775 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:40:07.808757    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d\": container with ID starting with 0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d not found: ID does not exist" containerID="0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"
Sep 07 10:40:07.808836 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.808783    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d"} err="failed to get container status \"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d\": rpc error: code = NotFound desc = could not find container \"0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d\": container with ID starting with 0ca0112742bf9e2eedac91c1e2476c2921da9a3364320b50bfed44011da7b73d not found: ID does not exist"
Sep 07 10:40:07.820052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.820032    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:40:07.828148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:07.828129    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6cd248c6-5c6874d76b-c7w2k"]
Sep 07 10:40:08.496585 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:08.496544    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42344fac-7107-4151-b6c2-c4d196bc53e4" path="/var/lib/kubelet/pods/42344fac-7107-4151-b6c2-c4d196bc53e4/volumes"
Sep 07 10:40:12.472033 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.471998    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:40:12.472876 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.472854    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:40:12.523689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.523665    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:40:12.523995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.523981    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42344fac-7107-4151-b6c2-c4d196bc53e4" containerName="payload-processing"
Sep 07 10:40:12.524054 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.523998    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="42344fac-7107-4151-b6c2-c4d196bc53e4" containerName="payload-processing"
Sep 07 10:40:12.524090 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.524056    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="42344fac-7107-4151-b6c2-c4d196bc53e4" containerName="payload-processing"
Sep 07 10:40:12.527992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.527978    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.541340 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.534692    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-36798ca4-kserve-self-signed-certs\""
Sep 07 10:40:12.544591 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.544566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:40:12.639376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639348    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.639493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.639493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.639493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.639643 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.639643 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.639540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2sk\" (UniqueName: \"kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740159 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740159 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740182    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2sk\" (UniqueName: \"kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740376 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740589 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740589 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.740744 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.740623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.742577 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.742556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.742907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.742887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.750078 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.750058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2sk\" (UniqueName: \"kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk\") pod \"e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") " pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.837580 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.837552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:12.958080 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.958050    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:12.961473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.961458    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:12.964226 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.964203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc9185fe\""
Sep 07 10:40:12.964356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.964328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc9185fe-dockercfg-kl4f6\""
Sep 07 10:40:12.972160 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:12.972136    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:13.042565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.042489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkfk\" (UniqueName: \"kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.042565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.042560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.109437 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.109419    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:40:13.111445 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:40:13.111411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2982b00b_f915_438a_ac50_c361e6753d1f.slice/crio-11ade6d4be02c0c4c3bfe2f74cba9c1807e5038e41ed4a8268499654cc4ee6dc WatchSource:0}: Error finding container 11ade6d4be02c0c4c3bfe2f74cba9c1807e5038e41ed4a8268499654cc4ee6dc: Status 404 returned error can't find the container with id 11ade6d4be02c0c4c3bfe2f74cba9c1807e5038e41ed4a8268499654cc4ee6dc
Sep 07 10:40:13.143761 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.143740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkfk\" (UniqueName: \"kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.143839 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.143800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.144413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.144397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.153110 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.153090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkfk\" (UniqueName: \"kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk\") pod \"payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.272897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.272867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.553509 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.553483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:13.555127 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:40:13.555102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f74c71_1dd6_4c82_94a2_f7f39493cbe4.slice/crio-94ab2abcdc28db543ab582a1c410e17ae68419096c82bc5cfb1f2ddc45a5f5a9 WatchSource:0}: Error finding container 94ab2abcdc28db543ab582a1c410e17ae68419096c82bc5cfb1f2ddc45a5f5a9: Status 404 returned error can't find the container with id 94ab2abcdc28db543ab582a1c410e17ae68419096c82bc5cfb1f2ddc45a5f5a9
Sep 07 10:40:13.827101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.827006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" event={"ID":"86f74c71-1dd6-4c82-94a2-f7f39493cbe4","Type":"ContainerStarted","Data":"ab472b6b842dff8021229dc921f9fa8d43372b9ba7ccb7098c64ce54561d0c40"}
Sep 07 10:40:13.827101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.827076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:13.827101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.827100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" event={"ID":"86f74c71-1dd6-4c82-94a2-f7f39493cbe4","Type":"ContainerStarted","Data":"94ab2abcdc28db543ab582a1c410e17ae68419096c82bc5cfb1f2ddc45a5f5a9"}
Sep 07 10:40:13.828568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.828541    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" event={"ID":"2982b00b-f915-438a-ac50-c361e6753d1f","Type":"ContainerStarted","Data":"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"}
Sep 07 10:40:13.828682 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.828580    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" event={"ID":"2982b00b-f915-438a-ac50-c361e6753d1f","Type":"ContainerStarted","Data":"11ade6d4be02c0c4c3bfe2f74cba9c1807e5038e41ed4a8268499654cc4ee6dc"}
Sep 07 10:40:13.846428 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.846384    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" podStartSLOduration=1.846368487 podStartE2EDuration="1.846368487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:13.845080159 +0000 UTC m=+1201.855733478" watchObservedRunningTime="2026-09-07 10:40:13.846368487 +0000 UTC m=+1201.857021845"
Sep 07 10:40:13.863071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:13.863037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" podStartSLOduration=1.863024035 podStartE2EDuration="1.863024035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:13.861852741 +0000 UTC m=+1201.872506056" watchObservedRunningTime="2026-09-07 10:40:13.863024035 +0000 UTC m=+1201.873677349"
Sep 07 10:40:22.838237 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:22.838205    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:22.838826 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:22.838257    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:22.850068 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:22.850048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:22.877553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:22.877531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:40:24.834758 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:24.834729    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:30.267259 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.267225    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:40:30.267702 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.267475    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" podUID="07e9e650-573e-41c9-a155-52a0720a1ce9" containerName="payload-processing" containerID="cri-o://0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809" gracePeriod=30
Sep 07 10:40:30.519487 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.519431    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:40:30.686606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.686576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w58ql\" (UniqueName: \"kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql\") pod \"07e9e650-573e-41c9-a155-52a0720a1ce9\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") "
Sep 07 10:40:30.686606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.686612    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume\") pod \"07e9e650-573e-41c9-a155-52a0720a1ce9\" (UID: \"07e9e650-573e-41c9-a155-52a0720a1ce9\") "
Sep 07 10:40:30.686991 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.686970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07e9e650-573e-41c9-a155-52a0720a1ce9" (UID: "07e9e650-573e-41c9-a155-52a0720a1ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:30.688770 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.688750    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql" (OuterVolumeSpecName: "kube-api-access-w58ql") pod "07e9e650-573e-41c9-a155-52a0720a1ce9" (UID: "07e9e650-573e-41c9-a155-52a0720a1ce9"). InnerVolumeSpecName "kube-api-access-w58ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:30.788097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.788008    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58ql\" (UniqueName: \"kubernetes.io/projected/07e9e650-573e-41c9-a155-52a0720a1ce9-kube-api-access-w58ql\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:30.788097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.788048    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e9e650-573e-41c9-a155-52a0720a1ce9-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:30.902255 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.902221    2562 generic.go:356] "Generic (PLEG): container finished" podID="07e9e650-573e-41c9-a155-52a0720a1ce9" containerID="0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809" exitCode=0
Sep 07 10:40:30.902465 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.902262    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"
Sep 07 10:40:30.902465 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.902319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" event={"ID":"07e9e650-573e-41c9-a155-52a0720a1ce9","Type":"ContainerDied","Data":"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"}
Sep 07 10:40:30.902465 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.902375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx" event={"ID":"07e9e650-573e-41c9-a155-52a0720a1ce9","Type":"ContainerDied","Data":"0f310a379187af7625b228cad74cf5358137b7937995d47196c5a77abf16d5fe"}
Sep 07 10:40:30.902465 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.902398    2562 scope.go:117] "RemoveContainer" containerID="0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"
Sep 07 10:40:30.913158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.913140    2562 scope.go:117] "RemoveContainer" containerID="0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"
Sep 07 10:40:30.913490 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:40:30.913473    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809\": container with ID starting with 0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809 not found: ID does not exist" containerID="0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"
Sep 07 10:40:30.913564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.913498    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809"} err="failed to get container status \"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809\": rpc error: code = NotFound desc = could not find container \"0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809\": container with ID starting with 0fd2916d2117ec5b0cf85c8b25ebcc65138ef74eee93798956f7fc78ababf809 not found: ID does not exist"
Sep 07 10:40:30.924178 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.924158    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:40:30.927610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:30.927582    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c5b29a-f9cbcb6fc-rj2rx"]
Sep 07 10:40:32.496310 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:32.496267    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07e9e650-573e-41c9-a155-52a0720a1ce9" path="/var/lib/kubelet/pods/07e9e650-573e-41c9-a155-52a0720a1ce9/volumes"
Sep 07 10:40:36.018253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:36.018223    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:40:42.233394 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.233363    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:40:42.233878 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.233859    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07e9e650-573e-41c9-a155-52a0720a1ce9" containerName="payload-processing"
Sep 07 10:40:42.233921 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.233882    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07e9e650-573e-41c9-a155-52a0720a1ce9" containerName="payload-processing"
Sep 07 10:40:42.233994 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.233982    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07e9e650-573e-41c9-a155-52a0720a1ce9" containerName="payload-processing"
Sep 07 10:40:42.236839 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.236825    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.239561 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.239540    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1ef687\"/\"openshift-service-ca.crt\""
Sep 07 10:40:42.239676 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.239544    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1ef687\"/\"default-dockercfg-jtjsm\""
Sep 07 10:40:42.240514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.240493    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1ef687\"/\"kube-root-ca.crt\""
Sep 07 10:40:42.240607 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.240516    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1ef687\"/\"test-model-1ef687-kserve-self-signed-certs\""
Sep 07 10:40:42.244885 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.244867    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:40:42.287540 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.287666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8582v\" (UniqueName: \"kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.287666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.287666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.287782 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.287782 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.287687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8582v\" (UniqueName: \"kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388922 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.388981 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.388959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.389040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.389007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.390972 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.390945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.391090 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.391074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.396460 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.396441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8582v\" (UniqueName: \"kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v\") pod \"test-model-1ef687-kserve-5c78684bdc-cqjvs\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") " pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.546044 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.545957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:42.818336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.818291    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:40:42.820869 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:40:42.820821    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae17dedd_159d_4623_b624_3f190f29a4ea.slice/crio-c0c6d074215a5e0064b2b2114830e5a95d31ebb14f902efd5f2d93d01644bb56 WatchSource:0}: Error finding container c0c6d074215a5e0064b2b2114830e5a95d31ebb14f902efd5f2d93d01644bb56: Status 404 returned error can't find the container with id c0c6d074215a5e0064b2b2114830e5a95d31ebb14f902efd5f2d93d01644bb56
Sep 07 10:40:42.822466 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.822445    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:42.822656 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.822636    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" podUID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" containerName="payload-pre-processing" containerID="cri-o://ab472b6b842dff8021229dc921f9fa8d43372b9ba7ccb7098c64ce54561d0c40" gracePeriod=30
Sep 07 10:40:42.955737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.955710    2562 generic.go:356] "Generic (PLEG): container finished" podID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" containerID="ab472b6b842dff8021229dc921f9fa8d43372b9ba7ccb7098c64ce54561d0c40" exitCode=0
Sep 07 10:40:42.955856 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.955792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" event={"ID":"86f74c71-1dd6-4c82-94a2-f7f39493cbe4","Type":"ContainerDied","Data":"ab472b6b842dff8021229dc921f9fa8d43372b9ba7ccb7098c64ce54561d0c40"}
Sep 07 10:40:42.957121 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:42.957094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" event={"ID":"ae17dedd-159d-4623-b624-3f190f29a4ea","Type":"ContainerStarted","Data":"c0c6d074215a5e0064b2b2114830e5a95d31ebb14f902efd5f2d93d01644bb56"}
Sep 07 10:40:43.069433 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.069374    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:43.094312 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.094273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bkfk\" (UniqueName: \"kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk\") pod \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") "
Sep 07 10:40:43.094444 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.094365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume\") pod \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\" (UID: \"86f74c71-1dd6-4c82-94a2-f7f39493cbe4\") "
Sep 07 10:40:43.094757 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.094728    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86f74c71-1dd6-4c82-94a2-f7f39493cbe4" (UID: "86f74c71-1dd6-4c82-94a2-f7f39493cbe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:43.096338 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.096290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk" (OuterVolumeSpecName: "kube-api-access-9bkfk") pod "86f74c71-1dd6-4c82-94a2-f7f39493cbe4" (UID: "86f74c71-1dd6-4c82-94a2-f7f39493cbe4"). InnerVolumeSpecName "kube-api-access-9bkfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:43.195442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.195410    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:43.195442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.195434    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bkfk\" (UniqueName: \"kubernetes.io/projected/86f74c71-1dd6-4c82-94a2-f7f39493cbe4-kube-api-access-9bkfk\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:43.962924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.962897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"
Sep 07 10:40:43.963400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.962895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2" event={"ID":"86f74c71-1dd6-4c82-94a2-f7f39493cbe4","Type":"ContainerDied","Data":"94ab2abcdc28db543ab582a1c410e17ae68419096c82bc5cfb1f2ddc45a5f5a9"}
Sep 07 10:40:43.963400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.963023    2562 scope.go:117] "RemoveContainer" containerID="ab472b6b842dff8021229dc921f9fa8d43372b9ba7ccb7098c64ce54561d0c40"
Sep 07 10:40:43.964794 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.964768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" event={"ID":"ae17dedd-159d-4623-b624-3f190f29a4ea","Type":"ContainerStarted","Data":"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"}
Sep 07 10:40:43.985554 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:43.985512    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" podStartSLOduration=1.985496205 podStartE2EDuration="1.985496205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:43.984543819 +0000 UTC m=+1231.995197133" watchObservedRunningTime="2026-09-07 10:40:43.985496205 +0000 UTC m=+1231.996149567"
Sep 07 10:40:44.000713 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:44.000682    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:44.004531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:44.004511    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9185fe-fd5f5dcdb-b85g2"]
Sep 07 10:40:44.496064 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:44.496030    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" path="/var/lib/kubelet/pods/86f74c71-1dd6-4c82-94a2-f7f39493cbe4/volumes"
Sep 07 10:40:51.038195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.038153    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" podUID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" containerName="main" containerID="cri-o://7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e" gracePeriod=45
Sep 07 10:40:51.281614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.281589    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:40:51.356966 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.356940    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.356996    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357014    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357050    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf7nr\" (UniqueName: \"kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr\") pod \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\" (UID: \"a34e0f14-f561-4fc1-a8c3-b52a22df764b\") "
Sep 07 10:40:51.357385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:51.357385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357345    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home" (OuterVolumeSpecName: "home") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:51.357491 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.357380    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:51.359150 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.359128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm" (OuterVolumeSpecName: "dshm") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:51.359277 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.359254    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:40:51.359358 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.359339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr" (OuterVolumeSpecName: "kube-api-access-jf7nr") pod "a34e0f14-f561-4fc1-a8c3-b52a22df764b" (UID: "a34e0f14-f561-4fc1-a8c3-b52a22df764b"). InnerVolumeSpecName "kube-api-access-jf7nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:51.458520 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458500    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:51.458520 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458518    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:51.458637 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458526    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:51.458637 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458534    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:51.458637 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458541    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a34e0f14-f561-4fc1-a8c3-b52a22df764b-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:51.458637 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:51.458549    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7nr\" (UniqueName: \"kubernetes.io/projected/a34e0f14-f561-4fc1-a8c3-b52a22df764b-kube-api-access-jf7nr\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:40:52.001848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.001814    2562 generic.go:356] "Generic (PLEG): container finished" podID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" containerID="7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e" exitCode=0
Sep 07 10:40:52.002014 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.001857    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"
Sep 07 10:40:52.002014 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.001899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" event={"ID":"a34e0f14-f561-4fc1-a8c3-b52a22df764b","Type":"ContainerDied","Data":"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"}
Sep 07 10:40:52.002014 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.001944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h" event={"ID":"a34e0f14-f561-4fc1-a8c3-b52a22df764b","Type":"ContainerDied","Data":"280fae58a5b74abbe2c0163935daecee7660bd6c23abf07ca51fd5b32b078660"}
Sep 07 10:40:52.002014 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.001963    2562 scope.go:117] "RemoveContainer" containerID="7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"
Sep 07 10:40:52.012285 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.012266    2562 scope.go:117] "RemoveContainer" containerID="7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"
Sep 07 10:40:52.012593 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:40:52.012567    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e\": container with ID starting with 7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e not found: ID does not exist" containerID="7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"
Sep 07 10:40:52.012657 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.012607    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e"} err="failed to get container status \"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e\": rpc error: code = NotFound desc = could not find container \"7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e\": container with ID starting with 7c56b653afa1d6f35fcbde0c39165ffb9ade7c4e2b143a3b6e73f4dd9164222e not found: ID does not exist"
Sep 07 10:40:52.025862 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.025841    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:40:52.029441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.029422    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c5b29a/autores-model-c5b29a-kserve-76c7c6ff99-5mx8h"]
Sep 07 10:40:52.497087 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.497055    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" path="/var/lib/kubelet/pods/a34e0f14-f561-4fc1-a8c3-b52a22df764b/volumes"
Sep 07 10:40:52.546550 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.546529    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:52.546628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.546573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:52.558872 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:52.558847    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:53.017715 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:53.017680    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:40:54.517044 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:40:54.517009    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:41:09.536610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.536571    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" podUID="2982b00b-f915-438a-ac50-c361e6753d1f" containerName="main" containerID="cri-o://e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a" gracePeriod=45
Sep 07 10:41:09.785997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.785974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:41:09.917813 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.917787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.917969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.917834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp2sk\" (UniqueName: \"kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.917969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.917874    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.917969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.917893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.917969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.917916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.918173 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918041    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:09.918173 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home\") pod \"2982b00b-f915-438a-ac50-c361e6753d1f\" (UID: \"2982b00b-f915-438a-ac50-c361e6753d1f\") "
Sep 07 10:41:09.918349 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918328    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:09.918444 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918418    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home" (OuterVolumeSpecName: "home") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:09.918564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918521    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:09.918564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918539    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:09.918652 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.918561    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:09.920220 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.920195    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:09.920443 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.920420    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm" (OuterVolumeSpecName: "dshm") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:09.920514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:09.920434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk" (OuterVolumeSpecName: "kube-api-access-rp2sk") pod "2982b00b-f915-438a-ac50-c361e6753d1f" (UID: "2982b00b-f915-438a-ac50-c361e6753d1f"). InnerVolumeSpecName "kube-api-access-rp2sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:10.019561 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.019538    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2982b00b-f915-438a-ac50-c361e6753d1f-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:10.019561 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.019559    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2sk\" (UniqueName: \"kubernetes.io/projected/2982b00b-f915-438a-ac50-c361e6753d1f-kube-api-access-rp2sk\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:10.019686 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.019569    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2982b00b-f915-438a-ac50-c361e6753d1f-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:10.086083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.086060    2562 generic.go:356] "Generic (PLEG): container finished" podID="2982b00b-f915-438a-ac50-c361e6753d1f" containerID="e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a" exitCode=0
Sep 07 10:41:10.086191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.086094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" event={"ID":"2982b00b-f915-438a-ac50-c361e6753d1f","Type":"ContainerDied","Data":"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"}
Sep 07 10:41:10.086191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.086103    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"
Sep 07 10:41:10.086191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.086121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq" event={"ID":"2982b00b-f915-438a-ac50-c361e6753d1f","Type":"ContainerDied","Data":"11ade6d4be02c0c4c3bfe2f74cba9c1807e5038e41ed4a8268499654cc4ee6dc"}
Sep 07 10:41:10.086191 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.086144    2562 scope.go:117] "RemoveContainer" containerID="e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"
Sep 07 10:41:10.094736 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.094720    2562 scope.go:117] "RemoveContainer" containerID="e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"
Sep 07 10:41:10.094976 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:41:10.094960    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a\": container with ID starting with e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a not found: ID does not exist" containerID="e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"
Sep 07 10:41:10.095025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.094983    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a"} err="failed to get container status \"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a\": rpc error: code = NotFound desc = could not find container \"e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a\": container with ID starting with e86055ee903f2ff99452d768cbcc89001a541e64fc4b36dd2a1ecbb81a53a81a not found: ID does not exist"
Sep 07 10:41:10.106949 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.106924    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:41:10.112406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.112387    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-36798ca4-kserve-7ffb697448-zbvpq"]
Sep 07 10:41:10.497991 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:10.497959    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2982b00b-f915-438a-ac50-c361e6753d1f" path="/var/lib/kubelet/pods/2982b00b-f915-438a-ac50-c361e6753d1f/volumes"
Sep 07 10:41:11.112775 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.112737    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113019    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" containerName="main"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113030    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" containerName="main"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113042    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2982b00b-f915-438a-ac50-c361e6753d1f" containerName="main"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113047    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2982b00b-f915-438a-ac50-c361e6753d1f" containerName="main"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113057    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" containerName="payload-pre-processing"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113062    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" containerName="payload-pre-processing"
Sep 07 10:41:11.113142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113140    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2982b00b-f915-438a-ac50-c361e6753d1f" containerName="main"
Sep 07 10:41:11.113425 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113156    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f74c71-1dd6-4c82-94a2-f7f39493cbe4" containerName="payload-pre-processing"
Sep 07 10:41:11.113425 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.113164    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34e0f14-f561-4fc1-a8c3-b52a22df764b" containerName="main"
Sep 07 10:41:11.117792 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.117767    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.121119 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.121097    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a49ef7-openshift-default-dockercfg-87xjj\""
Sep 07 10:41:11.132507 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.132486    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:41:11.229089 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229059    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229122    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh9sc\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.229442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.229294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330467 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330467 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh9sc\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330653    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330981 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330679    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330981 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.330981 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.330926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.331130 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.331000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.331130 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.331055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.331408 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.331387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.332996 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.332968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.333139 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.333120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.338853 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.338823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.338947 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.338930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh9sc\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc\") pod \"e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") " pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.428178 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.428121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:11.702416 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.702390    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:41:11.704461 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:41:11.704435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d2ebe8_476b_4baf_b717_a7fe2ed59598.slice/crio-599a1a8de851e1bbf16497fa09b3b0e4cfb8480c613d81479d203ef7f9ead0a9 WatchSource:0}: Error finding container 599a1a8de851e1bbf16497fa09b3b0e4cfb8480c613d81479d203ef7f9ead0a9: Status 404 returned error can't find the container with id 599a1a8de851e1bbf16497fa09b3b0e4cfb8480c613d81479d203ef7f9ead0a9
Sep 07 10:41:11.706843 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.706809    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:11.706947 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.706868    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:11.706947 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:11.706895    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:12.096588 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:12.096549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerStarted","Data":"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"}
Sep 07 10:41:12.096739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:12.096619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerStarted","Data":"599a1a8de851e1bbf16497fa09b3b0e4cfb8480c613d81479d203ef7f9ead0a9"}
Sep 07 10:41:12.117957 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:12.117908    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podStartSLOduration=1.11789106 podStartE2EDuration="1.11789106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:41:12.115667618 +0000 UTC m=+1260.126320933" watchObservedRunningTime="2026-09-07 10:41:12.11789106 +0000 UTC m=+1260.128544401"
Sep 07 10:41:12.428803 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:12.428731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:12.433867 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:12.433846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:13.100789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.100762    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:13.101540 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.101521    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:13.151701 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.151674    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"]
Sep 07 10:41:13.156464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.156440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:13.159292 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.159271    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:41:13.163378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.163197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"]
Sep 07 10:41:13.249137 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.249109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm\" (UID: \"6e880317-8999-49f8-b466-3f25cab1ca79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:13.349844 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.349810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm\" (UID: \"6e880317-8999-49f8-b466-3f25cab1ca79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:13.350466 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.350448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm\" (UID: \"6e880317-8999-49f8-b466-3f25cab1ca79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:13.467910 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.467848    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:13.738279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:13.738256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"]
Sep 07 10:41:13.740319 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:41:13.740276    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e880317_8999_49f8_b466_3f25cab1ca79.slice/crio-bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca WatchSource:0}: Error finding container bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca: Status 404 returned error can't find the container with id bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca
Sep 07 10:41:14.106263 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:14.106225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm" event={"ID":"6e880317-8999-49f8-b466-3f25cab1ca79","Type":"ContainerStarted","Data":"b5ab57ba24084cbe07f2038809188df8ad85844cf50b0087e462d86f792b212c"}
Sep 07 10:41:14.106463 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:14.106274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm" event={"ID":"6e880317-8999-49f8-b466-3f25cab1ca79","Type":"ContainerStarted","Data":"bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca"}
Sep 07 10:41:14.122342 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:14.122282    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm" podStartSLOduration=1.122268367 podStartE2EDuration="1.122268367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:41:14.120971189 +0000 UTC m=+1262.131624501" watchObservedRunningTime="2026-09-07 10:41:14.122268367 +0000 UTC m=+1262.132921734"
Sep 07 10:41:15.111323 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:15.111264    2562 generic.go:356] "Generic (PLEG): container finished" podID="6e880317-8999-49f8-b466-3f25cab1ca79" containerID="b5ab57ba24084cbe07f2038809188df8ad85844cf50b0087e462d86f792b212c" exitCode=0
Sep 07 10:41:15.111689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:15.111351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm" event={"ID":"6e880317-8999-49f8-b466-3f25cab1ca79","Type":"ContainerDied","Data":"b5ab57ba24084cbe07f2038809188df8ad85844cf50b0087e462d86f792b212c"}
Sep 07 10:41:16.001792 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.001751    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:41:16.005112 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.005090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.007957 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.007937    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a49ef7\""
Sep 07 10:41:16.008085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.007945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a49ef7-dockercfg-7g7ck\""
Sep 07 10:41:16.014968 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.014944    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:41:16.086313 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.086280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhkx\" (UniqueName: \"kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.086462 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.086433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.187186 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.187126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.187716 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.187233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhkx\" (UniqueName: \"kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.188674 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.188647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.196066 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.196039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhkx\" (UniqueName: \"kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx\") pod \"payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") " pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.242478 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.242458    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:16.316081 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.316055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:16.389214 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.389181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca\") pod \"6e880317-8999-49f8-b466-3f25cab1ca79\" (UID: \"6e880317-8999-49f8-b466-3f25cab1ca79\") "
Sep 07 10:41:16.389534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.389509    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e880317-8999-49f8-b466-3f25cab1ca79" (UID: "6e880317-8999-49f8-b466-3f25cab1ca79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:16.490641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.490607    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e880317-8999-49f8-b466-3f25cab1ca79-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:16.514257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.514231    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:41:16.589494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:16.589471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:41:16.591649 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:41:16.591624    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3befd2_7fbf_4cce_aa4c_f469b104f7c1.slice/crio-15906cfd5d706d29092807749d6b23a59da52d2cfa8ce67c935d10abe0a9a723 WatchSource:0}: Error finding container 15906cfd5d706d29092807749d6b23a59da52d2cfa8ce67c935d10abe0a9a723: Status 404 returned error can't find the container with id 15906cfd5d706d29092807749d6b23a59da52d2cfa8ce67c935d10abe0a9a723
Sep 07 10:41:17.123636 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.123602    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"
Sep 07 10:41:17.123820 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.123605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm" event={"ID":"6e880317-8999-49f8-b466-3f25cab1ca79","Type":"ContainerDied","Data":"bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca"}
Sep 07 10:41:17.123820 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.123722    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb092e0d0fed097bf1bb9337fa8ef580febeac2c5daf42251e20ac020344c5ca"
Sep 07 10:41:17.125623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.125467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" event={"ID":"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1","Type":"ContainerStarted","Data":"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"}
Sep 07 10:41:17.125623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.125526    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:17.125623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.125551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" event={"ID":"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1","Type":"ContainerStarted","Data":"15906cfd5d706d29092807749d6b23a59da52d2cfa8ce67c935d10abe0a9a723"}
Sep 07 10:41:17.157716 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:17.157664    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" podStartSLOduration=2.157647706 podStartE2EDuration="2.157647706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:41:17.157178521 +0000 UTC m=+1265.167831836" watchObservedRunningTime="2026-09-07 10:41:17.157647706 +0000 UTC m=+1265.168301020"
Sep 07 10:41:20.382552 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:20.382519    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"]
Sep 07 10:41:20.384589 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:20.384563    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1ccb3bcb-2l8lm"]
Sep 07 10:41:20.495727 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:20.495698    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e880317-8999-49f8-b466-3f25cab1ca79" path="/var/lib/kubelet/pods/6e880317-8999-49f8-b466-3f25cab1ca79/volumes"
Sep 07 10:41:28.131429 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:28.131395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:41:31.533384 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.533336    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" podUID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" containerName="main" containerID="cri-o://522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f" gracePeriod=45
Sep 07 10:41:31.765865 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.765844    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:41:31.837387 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mfvt\" (UniqueName: \"kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837501 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837501 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837432    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837501 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837528    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs\") pod \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\" (UID: \"dfe0d624-2bcb-4ac6-866c-21f6748ce557\") "
Sep 07 10:41:31.837746 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837657    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:31.837849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:31.837958 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837888    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home" (OuterVolumeSpecName: "home") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:31.837958 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.837899    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:31.839533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.839511    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt" (OuterVolumeSpecName: "kube-api-access-2mfvt") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "kube-api-access-2mfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:31.839738 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.839714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm" (OuterVolumeSpecName: "dshm") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:41:31.840041 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.840020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfe0d624-2bcb-4ac6-866c-21f6748ce557" (UID: "dfe0d624-2bcb-4ac6-866c-21f6748ce557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:31.938689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.938666    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:31.938689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.938688    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:31.938824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.938696    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:31.938824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.938707    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfvt\" (UniqueName: \"kubernetes.io/projected/dfe0d624-2bcb-4ac6-866c-21f6748ce557-kube-api-access-2mfvt\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:31.938824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:31.938715    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfe0d624-2bcb-4ac6-866c-21f6748ce557-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:41:32.190217 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.190131    2562 generic.go:356] "Generic (PLEG): container finished" podID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" containerID="522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f" exitCode=0
Sep 07 10:41:32.190217 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.190182    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"
Sep 07 10:41:32.190434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.190223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" event={"ID":"dfe0d624-2bcb-4ac6-866c-21f6748ce557","Type":"ContainerDied","Data":"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"}
Sep 07 10:41:32.190434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.190265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p" event={"ID":"dfe0d624-2bcb-4ac6-866c-21f6748ce557","Type":"ContainerDied","Data":"3b512822866c55c0c26101ff04c10f863e464fa810a28d0a9dfd2d56730a0074"}
Sep 07 10:41:32.190434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.190281    2562 scope.go:117] "RemoveContainer" containerID="522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"
Sep 07 10:41:32.199272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.199251    2562 scope.go:117] "RemoveContainer" containerID="522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"
Sep 07 10:41:32.199662 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:41:32.199633    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f\": container with ID starting with 522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f not found: ID does not exist" containerID="522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"
Sep 07 10:41:32.199777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.199673    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f"} err="failed to get container status \"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f\": rpc error: code = NotFound desc = could not find container \"522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f\": container with ID starting with 522a925c8f20e4733708c7e05a7e7a8e09683a14d0d97d7d7b0d04fe2ae3252f not found: ID does not exist"
Sep 07 10:41:32.223538 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.223506    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:41:32.227629 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.227586    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-36798ca4-kserve-5cc475d5c4-ldm6p"]
Sep 07 10:41:32.498123 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:32.498046    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" path="/var/lib/kubelet/pods/dfe0d624-2bcb-4ac6-866c-21f6748ce557/volumes"
Sep 07 10:41:35.205692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.205659    2562 generic.go:356] "Generic (PLEG): container finished" podID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerID="4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe" exitCode=0
Sep 07 10:41:35.206106 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.205728    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerDied","Data":"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"}
Sep 07 10:41:35.206155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.206109    2562 scope.go:117] "RemoveContainer" containerID="4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"
Sep 07 10:41:35.207155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.207129    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:35.207249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.207195    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:35.207249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:35.207226    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:41:36.214653 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:36.214611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerStarted","Data":"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"}
Sep 07 10:41:36.429047 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:36.429006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:36.429285 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:36.429261    2562 patch_prober.go:28] interesting pod/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 07 10:41:36.429369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:36.429351    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 07 10:41:37.429028 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:37.428989    2562 patch_prober.go:28] interesting pod/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 07 10:41:37.429422 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:37.429056    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 07 10:41:38.429264 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:38.429227    2562 patch_prober.go:28] interesting pod/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 07 10:41:38.429684 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:38.429281    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 07 10:41:39.429481 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:39.429440    2562 patch_prober.go:28] interesting pod/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 07 10:41:39.429861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:39.429501    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 07 10:41:40.428700 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:40.428660    2562 patch_prober.go:28] interesting pod/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 07 10:41:40.428883 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:40.428722    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 07 10:41:41.432328 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:41.432280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:41.432727 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:41.432583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:41.433334 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:41.433287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:41:43.134889 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.134829    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135122    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" containerName="main"
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135132    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" containerName="main"
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135150    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e880317-8999-49f8-b466-3f25cab1ca79" containerName="revoke-keys"
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135156    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e880317-8999-49f8-b466-3f25cab1ca79" containerName="revoke-keys"
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135206    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe0d624-2bcb-4ac6-866c-21f6748ce557" containerName="main"
Sep 07 10:41:43.135400 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.135216    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e880317-8999-49f8-b466-3f25cab1ca79" containerName="revoke-keys"
Sep 07 10:41:43.138000 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.137977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.140598 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.140573    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a49ef7\"/\"openshift-service-ca.crt\""
Sep 07 10:41:43.140715 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.140614    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a49ef7\"/\"kube-root-ca.crt\""
Sep 07 10:41:43.141826 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.141808    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a49ef7\"/\"default-dockercfg-cj5pm\""
Sep 07 10:41:43.141935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.141887    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a49ef7\"/\"explicit-model-a49ef7-kserve-self-signed-certs\""
Sep 07 10:41:43.145910 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.145889    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:41:43.239273 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.239273 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.239500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.239500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.239500 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl9xr\" (UniqueName: \"kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.239602 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.239518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.340904 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.340863    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.340904 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.340906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.340952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.340969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.341016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl9xr\" (UniqueName: \"kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341162 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.341043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.341479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341557 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.341501    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.341557 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.341533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.343316 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.343283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.343448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.343429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.348984 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.348963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl9xr\" (UniqueName: \"kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr\") pod \"explicit-model-a49ef7-kserve-57b684fdd5-bq6nw\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") " pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.448123 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.448027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:43.716881 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:43.716768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:41:43.719046 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:41:43.719018    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0d6069d_ef29_4ecd_9319_4c87acda3611.slice/crio-f51149b2cdeb2448322a69ae34ce3cc18bd19eb49546b50f33981e2b8031b4f8 WatchSource:0}: Error finding container f51149b2cdeb2448322a69ae34ce3cc18bd19eb49546b50f33981e2b8031b4f8: Status 404 returned error can't find the container with id f51149b2cdeb2448322a69ae34ce3cc18bd19eb49546b50f33981e2b8031b4f8
Sep 07 10:41:44.259820 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:44.259787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" event={"ID":"d0d6069d-ef29-4ecd-9319-4c87acda3611","Type":"ContainerStarted","Data":"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"}
Sep 07 10:41:44.259820 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:44.259825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" event={"ID":"d0d6069d-ef29-4ecd-9319-4c87acda3611","Type":"ContainerStarted","Data":"f51149b2cdeb2448322a69ae34ce3cc18bd19eb49546b50f33981e2b8031b4f8"}
Sep 07 10:41:44.277768 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:44.277727    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" podStartSLOduration=1.277713369 podStartE2EDuration="1.277713369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:41:44.276550941 +0000 UTC m=+1292.287204255" watchObservedRunningTime="2026-09-07 10:41:44.277713369 +0000 UTC m=+1292.288366683"
Sep 07 10:41:53.448374 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:53.448341    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:53.448778 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:53.448697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:53.460521 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:53.460496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:54.317514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:54.317482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:41:59.295784 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:41:59.295753    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:42:09.941609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:09.941577    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:42:09.942019 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:09.941777    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" podUID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" containerName="payload-processing" containerID="cri-o://509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a" gracePeriod=30
Sep 07 10:42:10.181505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.181470    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:42:10.363963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.363930    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume\") pod \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") "
Sep 07 10:42:10.364153 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.364047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhhkx\" (UniqueName: \"kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx\") pod \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\" (UID: \"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1\") "
Sep 07 10:42:10.364378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.364351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" (UID: "7d3befd2-7fbf-4cce-aa4c-f469b104f7c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:42:10.366379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.366353    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx" (OuterVolumeSpecName: "kube-api-access-vhhkx") pod "7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" (UID: "7d3befd2-7fbf-4cce-aa4c-f469b104f7c1"). InnerVolumeSpecName "kube-api-access-vhhkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:10.375604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.375580    2562 generic.go:356] "Generic (PLEG): container finished" podID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" containerID="509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a" exitCode=0
Sep 07 10:42:10.375697 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.375626    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"
Sep 07 10:42:10.375697 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.375658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" event={"ID":"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1","Type":"ContainerDied","Data":"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"}
Sep 07 10:42:10.375765 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.375713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7" event={"ID":"7d3befd2-7fbf-4cce-aa4c-f469b104f7c1","Type":"ContainerDied","Data":"15906cfd5d706d29092807749d6b23a59da52d2cfa8ce67c935d10abe0a9a723"}
Sep 07 10:42:10.375765 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.375737    2562 scope.go:117] "RemoveContainer" containerID="509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"
Sep 07 10:42:10.384426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.384410    2562 scope.go:117] "RemoveContainer" containerID="509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"
Sep 07 10:42:10.384741 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:10.384723    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a\": container with ID starting with 509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a not found: ID does not exist" containerID="509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"
Sep 07 10:42:10.384801 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.384749    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a"} err="failed to get container status \"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a\": rpc error: code = NotFound desc = could not find container \"509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a\": container with ID starting with 509b285240321cad15ef4f86c3e8a4a3a94c22b88a597b24d9e0c559757e202a not found: ID does not exist"
Sep 07 10:42:10.398010 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.397988    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:42:10.401767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.401744    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a49ef7-b874c7d65-2pts7"]
Sep 07 10:42:10.465675 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.465640    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhkx\" (UniqueName: \"kubernetes.io/projected/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-kube-api-access-vhhkx\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:10.465675 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.465671    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:10.496075 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:10.496046    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" path="/var/lib/kubelet/pods/7d3befd2-7fbf-4cce-aa4c-f469b104f7c1/volumes"
Sep 07 10:42:14.315927 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.315887    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" podUID="d0d6069d-ef29-4ecd-9319-4c87acda3611" containerName="main" containerID="cri-o://fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba" gracePeriod=45
Sep 07 10:42:14.572603 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.572550    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:42:14.701454 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701425    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701515    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701539    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl9xr\" (UniqueName: \"kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701622    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home\") pod \"d0d6069d-ef29-4ecd-9319-4c87acda3611\" (UID: \"d0d6069d-ef29-4ecd-9319-4c87acda3611\") "
Sep 07 10:42:14.701874 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:14.701874 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701785    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:14.701874 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home" (OuterVolumeSpecName: "home") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:14.702023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701921    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:14.702023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701936    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:14.702023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.701951    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:14.703929 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.703907    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm" (OuterVolumeSpecName: "dshm") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:14.704022 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.703929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr" (OuterVolumeSpecName: "kube-api-access-cl9xr") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "kube-api-access-cl9xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:14.704022 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.703942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0d6069d-ef29-4ecd-9319-4c87acda3611" (UID: "d0d6069d-ef29-4ecd-9319-4c87acda3611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:42:14.802916 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.802880    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0d6069d-ef29-4ecd-9319-4c87acda3611-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:14.802916 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.802917    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl9xr\" (UniqueName: \"kubernetes.io/projected/d0d6069d-ef29-4ecd-9319-4c87acda3611-kube-api-access-cl9xr\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:14.803142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:14.802935    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0d6069d-ef29-4ecd-9319-4c87acda3611-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:15.399525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.399486    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0d6069d-ef29-4ecd-9319-4c87acda3611" containerID="fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba" exitCode=0
Sep 07 10:42:15.399525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.399528    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"
Sep 07 10:42:15.400063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.399542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" event={"ID":"d0d6069d-ef29-4ecd-9319-4c87acda3611","Type":"ContainerDied","Data":"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"}
Sep 07 10:42:15.400063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.399590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw" event={"ID":"d0d6069d-ef29-4ecd-9319-4c87acda3611","Type":"ContainerDied","Data":"f51149b2cdeb2448322a69ae34ce3cc18bd19eb49546b50f33981e2b8031b4f8"}
Sep 07 10:42:15.400063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.399616    2562 scope.go:117] "RemoveContainer" containerID="fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"
Sep 07 10:42:15.409239 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.409219    2562 scope.go:117] "RemoveContainer" containerID="fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"
Sep 07 10:42:15.409584 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:15.409556    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba\": container with ID starting with fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba not found: ID does not exist" containerID="fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"
Sep 07 10:42:15.409657 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.409594    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba"} err="failed to get container status \"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba\": rpc error: code = NotFound desc = could not find container \"fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba\": container with ID starting with fb5f7be958ed563eaea3358c9b8858a1d16ae91e17fc61effd5a1644bbb8f8ba not found: ID does not exist"
Sep 07 10:42:15.423116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.423091    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:42:15.425534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:15.425514    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a49ef7/explicit-model-a49ef7-kserve-57b684fdd5-bq6nw"]
Sep 07 10:42:16.495625 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:16.495594    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d6069d-ef29-4ecd-9319-4c87acda3611" path="/var/lib/kubelet/pods/d0d6069d-ef29-4ecd-9319-4c87acda3611/volumes"
Sep 07 10:42:16.625053 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:16.625021    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:42:16.625240 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:16.625216    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy" containerID="cri-o://a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28" gracePeriod=30
Sep 07 10:42:21.875600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.875571    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:42:21.959115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959070    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959097    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959115 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959116    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959177    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh9sc\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959263    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959291    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data\") pod \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\" (UID: \"58d2ebe8-476b-4baf-b717-a7fe2ed59598\") "
Sep 07 10:42:21.959448 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data" (OuterVolumeSpecName: "istio-data") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959647    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959652    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-credential-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:21.959735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.959700    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:21.961523 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.961495    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token" (OuterVolumeSpecName: "istio-token") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:21.961626 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.961522    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:21.961626 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.961531    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc" (OuterVolumeSpecName: "kube-api-access-zh9sc") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "kube-api-access-zh9sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:21.961626 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:21.961571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58d2ebe8-476b-4baf-b717-a7fe2ed59598" (UID: "58d2ebe8-476b-4baf-b717-a7fe2ed59598"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 10:42:22.061149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061116    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istiod-ca-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061145    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-workload-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061154    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh9sc\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-kube-api-access-zh9sc\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061399 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061164    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061399 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061173    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-data\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061399 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061180    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-podinfo\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.061399 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.061187    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58d2ebe8-476b-4baf-b717-a7fe2ed59598-istio-envoy\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:22.434505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.434472    2562 generic.go:356] "Generic (PLEG): container finished" podID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerID="a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28" exitCode=0
Sep 07 10:42:22.434688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.434525    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"
Sep 07 10:42:22.434688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.434553    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerDied","Data":"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"}
Sep 07 10:42:22.434688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.434608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb" event={"ID":"58d2ebe8-476b-4baf-b717-a7fe2ed59598","Type":"ContainerDied","Data":"599a1a8de851e1bbf16497fa09b3b0e4cfb8480c613d81479d203ef7f9ead0a9"}
Sep 07 10:42:22.434688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.434631    2562 scope.go:117] "RemoveContainer" containerID="a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"
Sep 07 10:42:22.443806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.443790    2562 scope.go:117] "RemoveContainer" containerID="4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"
Sep 07 10:42:22.452289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.452260    2562 scope.go:117] "RemoveContainer" containerID="a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"
Sep 07 10:42:22.452619 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:22.452598    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28\": container with ID starting with a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28 not found: ID does not exist" containerID="a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"
Sep 07 10:42:22.452675 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.452628    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28"} err="failed to get container status \"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28\": rpc error: code = NotFound desc = could not find container \"a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28\": container with ID starting with a6e71d6fc51f32d6a43d52d85efe8a44c859d098dc78e51de79694b71ddd3a28 not found: ID does not exist"
Sep 07 10:42:22.452675 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.452646    2562 scope.go:117] "RemoveContainer" containerID="4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"
Sep 07 10:42:22.452945 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:22.452926    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe\": container with ID starting with 4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe not found: ID does not exist" containerID="4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"
Sep 07 10:42:22.452986 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.452952    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe"} err="failed to get container status \"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe\": rpc error: code = NotFound desc = could not find container \"4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe\": container with ID starting with 4674e2e7ac2021a13e6dacebcad161cf7dc2221bea0a6cac91eebe6a47271ebe not found: ID does not exist"
Sep 07 10:42:22.456557 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.456529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:42:22.460554 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.460530    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a49ef7-openshift-default-77d46dbc88-x8qbb"]
Sep 07 10:42:22.498758 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:22.498722    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" path="/var/lib/kubelet/pods/58d2ebe8-476b-4baf-b717-a7fe2ed59598/volumes"
Sep 07 10:42:25.133093 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133044    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133540    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d6069d-ef29-4ecd-9319-4c87acda3611" containerName="main"
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133559    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d6069d-ef29-4ecd-9319-4c87acda3611" containerName="main"
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133583    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" containerName="payload-processing"
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133592    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" containerName="payload-processing"
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133613    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.133618 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133622    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133631    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133639    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133712    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d6069d-ef29-4ecd-9319-4c87acda3611" containerName="main"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133725    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133735    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d3befd2-7fbf-4cce-aa4c-f469b104f7c1" containerName="payload-processing"
Sep 07 10:42:25.133944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.133744    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d2ebe8-476b-4baf-b717-a7fe2ed59598" containerName="istio-proxy"
Sep 07 10:42:25.137130 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.137109    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.139936 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.139910    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2b5f38\"/\"kube-root-ca.crt\""
Sep 07 10:42:25.141519 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.141501    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2b5f38\"/\"default-dockercfg-gmgmz\""
Sep 07 10:42:25.141627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.141526    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2b5f38\"/\"noten-model-2b5f38-kserve-self-signed-certs\""
Sep 07 10:42:25.141627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.141543    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2b5f38\"/\"openshift-service-ca.crt\""
Sep 07 10:42:25.145855 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.145833    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:25.185718 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.185909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.185909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.185909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfd98\" (UniqueName: \"kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.185909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.186078 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.185902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287354 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfd98\" (UniqueName: \"kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287553 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287810 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287810 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.287810 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.287774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.289972 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.289941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.290118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.290102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.297963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.297942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfd98\" (UniqueName: \"kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98\") pod \"noten-model-2b5f38-kserve-684f65ffb7-x2ddb\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") " pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.447478 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.447378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:25.725251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:25.725228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:25.726853 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:42:25.726828    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9668d89c_f529_48ee_bc9a_ab158aea133e.slice/crio-0f37fe9d6b38266490fb5977e4a08791c4978f57d2764c03e7aae971a4996db4 WatchSource:0}: Error finding container 0f37fe9d6b38266490fb5977e4a08791c4978f57d2764c03e7aae971a4996db4: Status 404 returned error can't find the container with id 0f37fe9d6b38266490fb5977e4a08791c4978f57d2764c03e7aae971a4996db4
Sep 07 10:42:26.455772 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:26.455736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" event={"ID":"9668d89c-f529-48ee-bc9a-ab158aea133e","Type":"ContainerStarted","Data":"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"}
Sep 07 10:42:26.456247 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:26.455780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" event={"ID":"9668d89c-f529-48ee-bc9a-ab158aea133e","Type":"ContainerStarted","Data":"0f37fe9d6b38266490fb5977e4a08791c4978f57d2764c03e7aae971a4996db4"}
Sep 07 10:42:26.473661 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:26.473606    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" podStartSLOduration=1.473586156 podStartE2EDuration="1.473586156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:42:26.473497808 +0000 UTC m=+1334.484151137" watchObservedRunningTime="2026-09-07 10:42:26.473586156 +0000 UTC m=+1334.484239478"
Sep 07 10:42:35.447577 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:35.447481    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:35.448046 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:35.447770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:35.460234 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:35.460201    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:35.504953 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:35.504919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:36.215185 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:36.215153    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:42:37.107265 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:37.107229    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:37.591644 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:37.591606    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:37.591849 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:37.591692    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:42:38.09166972 +0000 UTC m=+1346.102323013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:38.095592 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:38.095558    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:38.095764 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:38.095626    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:42:39.095612015 +0000 UTC m=+1347.106265321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:39.102940 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:39.102908    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:39.103336 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:39.102971    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:42:41.102957485 +0000 UTC m=+1349.113610782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:41.123259 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:41.123223    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:41.123672 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:41.123319    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:42:45.123282965 +0000 UTC m=+1353.133936259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:45.160322 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:45.160275    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:45.160693 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:45.160371    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:42:53.160353615 +0000 UTC m=+1361.171006910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:51.234938 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.234896    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" podUID="ae17dedd-159d-4623-b624-3f190f29a4ea" containerName="main" containerID="cri-o://714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc" gracePeriod=45
Sep 07 10:42:51.480805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.480776    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:42:51.561224 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.561134    2562 generic.go:356] "Generic (PLEG): container finished" podID="ae17dedd-159d-4623-b624-3f190f29a4ea" containerID="714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc" exitCode=0
Sep 07 10:42:51.561224 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.561179    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"
Sep 07 10:42:51.561430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.561227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" event={"ID":"ae17dedd-159d-4623-b624-3f190f29a4ea","Type":"ContainerDied","Data":"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"}
Sep 07 10:42:51.561430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.561273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs" event={"ID":"ae17dedd-159d-4623-b624-3f190f29a4ea","Type":"ContainerDied","Data":"c0c6d074215a5e0064b2b2114830e5a95d31ebb14f902efd5f2d93d01644bb56"}
Sep 07 10:42:51.561430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.561319    2562 scope.go:117] "RemoveContainer" containerID="714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"
Sep 07 10:42:51.569350 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.569326    2562 scope.go:117] "RemoveContainer" containerID="714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"
Sep 07 10:42:51.569612 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:51.569595    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc\": container with ID starting with 714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc not found: ID does not exist" containerID="714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"
Sep 07 10:42:51.569675 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.569619    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc"} err="failed to get container status \"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc\": rpc error: code = NotFound desc = could not find container \"714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc\": container with ID starting with 714f708c34b99105e239069ad4aa84ecc44df38f8751aac95e5b515c6e1e04fc not found: ID does not exist"
Sep 07 10:42:51.619981 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.619949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8582v\" (UniqueName: \"kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620121 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.619995    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620121 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620015    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620121 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620047    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620287 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620287 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs\") pod \"ae17dedd-159d-4623-b624-3f190f29a4ea\" (UID: \"ae17dedd-159d-4623-b624-3f190f29a4ea\") "
Sep 07 10:42:51.620287 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620275    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home" (OuterVolumeSpecName: "home") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:51.620468 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:51.620468 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:51.620612 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620596    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.620671 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620614    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.620671 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.620623    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.622430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.622405    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm" (OuterVolumeSpecName: "dshm") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:51.622521 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.622480    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v" (OuterVolumeSpecName: "kube-api-access-8582v") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "kube-api-access-8582v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:51.622521 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.622491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae17dedd-159d-4623-b624-3f190f29a4ea" (UID: "ae17dedd-159d-4623-b624-3f190f29a4ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:42:51.721546 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.721512    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8582v\" (UniqueName: \"kubernetes.io/projected/ae17dedd-159d-4623-b624-3f190f29a4ea-kube-api-access-8582v\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.721546 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.721542    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae17dedd-159d-4623-b624-3f190f29a4ea-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.721546 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.721552    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae17dedd-159d-4623-b624-3f190f29a4ea-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:51.882210 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.882179    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:42:51.890763 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:51.890739    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1ef687/test-model-1ef687-kserve-5c78684bdc-cqjvs"]
Sep 07 10:42:52.496405 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:52.496371    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae17dedd-159d-4623-b624-3f190f29a4ea" path="/var/lib/kubelet/pods/ae17dedd-159d-4623-b624-3f190f29a4ea/volumes"
Sep 07 10:42:53.236995 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:53.236950    2562 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-self-signed-certs: secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:53.237207 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:53.237039    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs podName:9668d89c-f529-48ee-bc9a-ab158aea133e nodeName:}" failed. No retries permitted until 2026-09-07 10:43:09.237019363 +0000 UTC m=+1377.247672656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs") pod "noten-model-2b5f38-kserve-684f65ffb7-x2ddb" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e") : secret "noten-model-2b5f38-kserve-self-signed-certs" not found
Sep 07 10:42:53.528727 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.528615    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" podUID="9668d89c-f529-48ee-bc9a-ab158aea133e" containerName="main" containerID="cri-o://aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e" gracePeriod=45
Sep 07 10:42:53.774250 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.774226    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:53.943731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943731 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfd98\" (UniqueName: \"kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943759    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943885    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache\") pod \"9668d89c-f529-48ee-bc9a-ab158aea133e\" (UID: \"9668d89c-f529-48ee-bc9a-ab158aea133e\") "
Sep 07 10:42:53.943995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.943982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home" (OuterVolumeSpecName: "home") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:53.944252 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.944142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:53.944336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.944257    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:53.944336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.944274    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:53.944336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.944280    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:53.946167 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.946146    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm" (OuterVolumeSpecName: "dshm") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:42:53.946265 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.946190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:42:53.946265 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:53.946206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98" (OuterVolumeSpecName: "kube-api-access-rfd98") pod "9668d89c-f529-48ee-bc9a-ab158aea133e" (UID: "9668d89c-f529-48ee-bc9a-ab158aea133e"). InnerVolumeSpecName "kube-api-access-rfd98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:42:54.045270 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.045231    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfd98\" (UniqueName: \"kubernetes.io/projected/9668d89c-f529-48ee-bc9a-ab158aea133e-kube-api-access-rfd98\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:54.045270 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.045267    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9668d89c-f529-48ee-bc9a-ab158aea133e-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:54.045270 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.045285    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:54.045533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.045324    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9668d89c-f529-48ee-bc9a-ab158aea133e-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:42:54.576008 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.575976    2562 generic.go:356] "Generic (PLEG): container finished" podID="9668d89c-f529-48ee-bc9a-ab158aea133e" containerID="aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e" exitCode=0
Sep 07 10:42:54.576008 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.576011    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"
Sep 07 10:42:54.576533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.576053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" event={"ID":"9668d89c-f529-48ee-bc9a-ab158aea133e","Type":"ContainerDied","Data":"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"}
Sep 07 10:42:54.576533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.576093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb" event={"ID":"9668d89c-f529-48ee-bc9a-ab158aea133e","Type":"ContainerDied","Data":"0f37fe9d6b38266490fb5977e4a08791c4978f57d2764c03e7aae971a4996db4"}
Sep 07 10:42:54.576533 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.576110    2562 scope.go:117] "RemoveContainer" containerID="aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"
Sep 07 10:42:54.584337 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.584295    2562 scope.go:117] "RemoveContainer" containerID="aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"
Sep 07 10:42:54.584639 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:42:54.584618    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e\": container with ID starting with aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e not found: ID does not exist" containerID="aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"
Sep 07 10:42:54.584698 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.584647    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e"} err="failed to get container status \"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e\": rpc error: code = NotFound desc = could not find container \"aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e\": container with ID starting with aa08f4456ea269c3fdc8745c26a7470fc32a1fd5d986b4f0227dde5ed50e433e not found: ID does not exist"
Sep 07 10:42:54.592438 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.592409    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:54.596040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:54.596010    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2b5f38/noten-model-2b5f38-kserve-684f65ffb7-x2ddb"]
Sep 07 10:42:56.495909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:42:56.495873    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9668d89c-f529-48ee-bc9a-ab158aea133e" path="/var/lib/kubelet/pods/9668d89c-f529-48ee-bc9a-ab158aea133e/volumes"
Sep 07 10:43:07.470489 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.470427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:43:07.471083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.470675    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerName="payload-pre-processing" containerID="cri-o://afa81fa7c0215c3b478dafca0349eaacd5f7289412ddc2ad94f0ea10af320dec" gracePeriod=30
Sep 07 10:43:07.503771 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.503732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.31:9004: connect: connection refused"
Sep 07 10:43:07.640683 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.640649    2562 generic.go:356] "Generic (PLEG): container finished" podID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerID="afa81fa7c0215c3b478dafca0349eaacd5f7289412ddc2ad94f0ea10af320dec" exitCode=0
Sep 07 10:43:07.640864 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.640702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" event={"ID":"1dfaa093-66b5-4ee4-8029-ecd56cf4d314","Type":"ContainerDied","Data":"afa81fa7c0215c3b478dafca0349eaacd5f7289412ddc2ad94f0ea10af320dec"}
Sep 07 10:43:07.727954 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.727894    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:43:07.761197 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.761157    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2bjg\" (UniqueName: \"kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg\") pod \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") "
Sep 07 10:43:07.761399 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.761258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume\") pod \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\" (UID: \"1dfaa093-66b5-4ee4-8029-ecd56cf4d314\") "
Sep 07 10:43:07.761756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.761727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dfaa093-66b5-4ee4-8029-ecd56cf4d314" (UID: "1dfaa093-66b5-4ee4-8029-ecd56cf4d314"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:43:07.763370 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.763342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg" (OuterVolumeSpecName: "kube-api-access-z2bjg") pod "1dfaa093-66b5-4ee4-8029-ecd56cf4d314" (UID: "1dfaa093-66b5-4ee4-8029-ecd56cf4d314"). InnerVolumeSpecName "kube-api-access-z2bjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:43:07.862154 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.862111    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bjg\" (UniqueName: \"kubernetes.io/projected/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-kube-api-access-z2bjg\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:07.862154 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:07.862146    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfaa093-66b5-4ee4-8029-ecd56cf4d314-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:08.645925 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:08.645899    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"
Sep 07 10:43:08.646441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:08.645924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz" event={"ID":"1dfaa093-66b5-4ee4-8029-ecd56cf4d314","Type":"ContainerDied","Data":"dfb6f215fc0ac5d364170a32862a104b0e37af627b3074d2e8bb2cf5f9c8bf03"}
Sep 07 10:43:08.646441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:08.645998    2562 scope.go:117] "RemoveContainer" containerID="afa81fa7c0215c3b478dafca0349eaacd5f7289412ddc2ad94f0ea10af320dec"
Sep 07 10:43:08.671225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:08.671192    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:43:08.673163 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:08.673139    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b4061f-7d4c9cdd74-4n9cz"]
Sep 07 10:43:10.495543 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:10.495505    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" path="/var/lib/kubelet/pods/1dfaa093-66b5-4ee4-8029-ecd56cf4d314/volumes"
Sep 07 10:43:14.183244 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:14.183210    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:43:14.183716 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:14.183436    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy" containerID="cri-o://0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2" gracePeriod=30
Sep 07 10:43:19.439956 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.439929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:43:19.573557 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573523    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573717 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb92v\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573717 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573597    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573799 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573799 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573799 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573860    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token\") pod \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\" (UID: \"6dd7ec01-4ad1-4c12-8482-014617e5fb6e\") "
Sep 07 10:43:19.573963 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:43:19.574161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573966    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:43:19.574161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.573967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:43:19.574161 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:43:19.574346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574198    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:43:19.574346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574239    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istiod-ca-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.574346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574258    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-credential-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.574346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574273    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-data\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.574346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.574287    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.576172 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.576142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:43:19.576172 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.576142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:43:19.576446 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.576424    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 10:43:19.576493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.576444    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v" (OuterVolumeSpecName: "kube-api-access-pb92v") pod "6dd7ec01-4ad1-4c12-8482-014617e5fb6e" (UID: "6dd7ec01-4ad1-4c12-8482-014617e5fb6e"). InnerVolumeSpecName "kube-api-access-pb92v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:43:19.675067 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.675036    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-workload-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.675067 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.675064    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.675067 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.675073    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb92v\" (UniqueName: \"kubernetes.io/projected/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-kube-api-access-pb92v\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.675334 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.675082    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-podinfo\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.675334 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.675091    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dd7ec01-4ad1-4c12-8482-014617e5fb6e-istio-envoy\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:43:19.697237 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.697200    2562 generic.go:356] "Generic (PLEG): container finished" podID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerID="0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2" exitCode=0
Sep 07 10:43:19.697426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.697275    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"
Sep 07 10:43:19.697426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.697286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerDied","Data":"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"}
Sep 07 10:43:19.697426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.697359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj" event={"ID":"6dd7ec01-4ad1-4c12-8482-014617e5fb6e","Type":"ContainerDied","Data":"a8b0fcfe7e98071da5db2fdaefebf88191602291d2a44648837bc62fe0434251"}
Sep 07 10:43:19.697426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.697385    2562 scope.go:117] "RemoveContainer" containerID="0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"
Sep 07 10:43:19.706187 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.706164    2562 scope.go:117] "RemoveContainer" containerID="3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"
Sep 07 10:43:19.714141 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.714124    2562 scope.go:117] "RemoveContainer" containerID="0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"
Sep 07 10:43:19.714430 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:43:19.714412    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2\": container with ID starting with 0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2 not found: ID does not exist" containerID="0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"
Sep 07 10:43:19.714487 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.714439    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2"} err="failed to get container status \"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2\": rpc error: code = NotFound desc = could not find container \"0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2\": container with ID starting with 0a9ef3167f4805abc3685b169348d350e71acc5a03eadcf71d74f08dd5a180f2 not found: ID does not exist"
Sep 07 10:43:19.714487 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.714456    2562 scope.go:117] "RemoveContainer" containerID="3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"
Sep 07 10:43:19.714715 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:43:19.714696    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296\": container with ID starting with 3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296 not found: ID does not exist" containerID="3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"
Sep 07 10:43:19.714757 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.714720    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296"} err="failed to get container status \"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296\": rpc error: code = NotFound desc = could not find container \"3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296\": container with ID starting with 3d2d3c283b9fccb198e442ca2804503e7d22e108d44aacf7a338f47c36026296 not found: ID does not exist"
Sep 07 10:43:19.719720 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.719692    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:43:19.723936 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:19.723910    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b4061f-openshift-default-759cc756bb-jhppj"]
Sep 07 10:43:20.495411 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.495374    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" path="/var/lib/kubelet/pods/6dd7ec01-4ad1-4c12-8482-014617e5fb6e/volumes"
Sep 07 10:43:20.599347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599312    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:43:20.599611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599599    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9668d89c-f529-48ee-bc9a-ab158aea133e" containerName="main"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599612    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9668d89c-f529-48ee-bc9a-ab158aea133e" containerName="main"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599624    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerName="payload-pre-processing"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599630    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerName="payload-pre-processing"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599641    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae17dedd-159d-4623-b624-3f190f29a4ea" containerName="main"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599646    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae17dedd-159d-4623-b624-3f190f29a4ea" containerName="main"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599655    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599661    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599668    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.599669 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599673    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.599942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599732    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dfaa093-66b5-4ee4-8029-ecd56cf4d314" containerName="payload-pre-processing"
Sep 07 10:43:20.599942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599739    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9668d89c-f529-48ee-bc9a-ab158aea133e" containerName="main"
Sep 07 10:43:20.599942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599746    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.599942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.599763    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae17dedd-159d-4623-b624-3f190f29a4ea" containerName="main"
Sep 07 10:43:20.609704 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.609673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.611442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.611418    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:43:20.612483 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.612461    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-5a0ff8\""
Sep 07 10:43:20.631369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.631333    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:43:20.631849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.631832    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd7ec01-4ad1-4c12-8482-014617e5fb6e" containerName="istio-proxy"
Sep 07 10:43:20.634839 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.634823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.638037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.638021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-5a0ff8-dockercfg-h2vsh\""
Sep 07 10:43:20.644867 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.644847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:43:20.681926 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.681892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kj2g\" (UniqueName: \"kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.681926 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.681930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78xf\" (UniqueName: \"kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.682139 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.681988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.682139 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.682045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.782628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.782533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.782628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.782599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.782877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.782642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kj2g\" (UniqueName: \"kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.782877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.782668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78xf\" (UniqueName: \"kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.783292 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.783271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.783357 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.783334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.791434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.791403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kj2g\" (UniqueName: \"kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g\") pod \"payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.791532 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.791445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78xf\" (UniqueName: \"kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf\") pod \"payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:20.920570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.920539    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:20.944554 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:20.944526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:21.191938 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.191877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:43:21.194108 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:43:21.194079    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd89136_5260_4d69_9621_482f55b85a9b.slice/crio-fad47d4b9fbc6c7773c975caa581cab040d2854066478cffe1bf4447f9940e24 WatchSource:0}: Error finding container fad47d4b9fbc6c7773c975caa581cab040d2854066478cffe1bf4447f9940e24: Status 404 returned error can't find the container with id fad47d4b9fbc6c7773c975caa581cab040d2854066478cffe1bf4447f9940e24
Sep 07 10:43:21.223587 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.223566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:43:21.226629 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:43:21.226607    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c55085a_9da3_4276_94ac_2542b3c5c39e.slice/crio-b4c0ea48d14aa8c18267f379786c46040c333e3294236f1602722576c5128b8d WatchSource:0}: Error finding container b4c0ea48d14aa8c18267f379786c46040c333e3294236f1602722576c5128b8d: Status 404 returned error can't find the container with id b4c0ea48d14aa8c18267f379786c46040c333e3294236f1602722576c5128b8d
Sep 07 10:43:21.709648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.709609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" event={"ID":"5c55085a-9da3-4276-94ac-2542b3c5c39e","Type":"ContainerStarted","Data":"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"}
Sep 07 10:43:21.710116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.709680    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:21.710116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.709702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" event={"ID":"5c55085a-9da3-4276-94ac-2542b3c5c39e","Type":"ContainerStarted","Data":"b4c0ea48d14aa8c18267f379786c46040c333e3294236f1602722576c5128b8d"}
Sep 07 10:43:21.711330 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.711283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" event={"ID":"6dd89136-5260-4d69-9621-482f55b85a9b","Type":"ContainerStarted","Data":"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"}
Sep 07 10:43:21.711457 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.711360    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:43:21.711457 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.711383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" event={"ID":"6dd89136-5260-4d69-9621-482f55b85a9b","Type":"ContainerStarted","Data":"fad47d4b9fbc6c7773c975caa581cab040d2854066478cffe1bf4447f9940e24"}
Sep 07 10:43:21.727042 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.726991    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" podStartSLOduration=1.726972252 podStartE2EDuration="1.726972252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:21.725149622 +0000 UTC m=+1389.735802940" watchObservedRunningTime="2026-09-07 10:43:21.726972252 +0000 UTC m=+1389.737625568"
Sep 07 10:43:21.741086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:21.741032    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" podStartSLOduration=1.741013099 podStartE2EDuration="1.741013099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:21.74062155 +0000 UTC m=+1389.751274866" watchObservedRunningTime="2026-09-07 10:43:21.741013099 +0000 UTC m=+1389.751666415"
Sep 07 10:43:32.718657 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:32.718628    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:43:32.719031 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:43:32.718952    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:44:05.972398 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:05.972361    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:44:05.976269 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:05.976248    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:05.979101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:05.979077    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a158df-openshift-default-dockercfg-jg4gq\""
Sep 07 10:44:05.979213 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:05.979095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 10:44:05.988983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:05.988959    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:44:06.071686 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071798 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071798 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxp7\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.071969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.072038 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.071966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172881 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172987 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172987 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172987 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172926    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172987 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxp7\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.172987 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.172986    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173217 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173217 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173359 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173359 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173333    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173463 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173520 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173623 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.173747 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.173711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.175471 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.175450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.175647 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.175627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.181218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.181191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.181326 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.181223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxp7\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7\") pod \"e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") " pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.290040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.289977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:06.565441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.565410    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:44:06.568436 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:44:06.568407    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fabef64_950d_4c7b_9c10_da0321281d9f.slice/crio-45e26468dc0782f23ff8b6b77ffde259085412ba738bb8a848100477f095494c WatchSource:0}: Error finding container 45e26468dc0782f23ff8b6b77ffde259085412ba738bb8a848100477f095494c: Status 404 returned error can't find the container with id 45e26468dc0782f23ff8b6b77ffde259085412ba738bb8a848100477f095494c
Sep 07 10:44:06.570579 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.570544    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:06.570692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.570608    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:06.570692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.570634    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:06.917156 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.917070    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerStarted","Data":"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"}
Sep 07 10:44:06.917156 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.917118    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerStarted","Data":"45e26468dc0782f23ff8b6b77ffde259085412ba738bb8a848100477f095494c"}
Sep 07 10:44:06.940709 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:06.940664    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podStartSLOduration=1.940648666 podStartE2EDuration="1.940648666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:06.938265089 +0000 UTC m=+1434.948918403" watchObservedRunningTime="2026-09-07 10:44:06.940648666 +0000 UTC m=+1434.951301981"
Sep 07 10:44:07.290328 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:07.290230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:07.295194 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:07.295168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:07.921823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:07.921785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:07.922630 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:07.922609    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:11.496025 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.495987    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:44:11.499119 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.499098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.501790 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.501766    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a158df-dockercfg-8tlq5\""
Sep 07 10:44:11.501790 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.501777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a158df\""
Sep 07 10:44:11.508167 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.508144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:44:11.621767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.621739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhbk\" (UniqueName: \"kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.621966 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.621858    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.723062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.723030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.723203 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.723084    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhbk\" (UniqueName: \"kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.723761 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.723744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.734668 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.734637    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhbk\" (UniqueName: \"kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk\") pod \"payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:11.810552 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:11.810487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:12.081979 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:12.081956    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:44:12.084359 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:44:12.084330    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958c82a1_332e_416d_9f03_7112419ca8f2.slice/crio-94d175a7a37c5c3ef8c10fc973b829f6b7a34e34bd1119637cfecac9941cecae WatchSource:0}: Error finding container 94d175a7a37c5c3ef8c10fc973b829f6b7a34e34bd1119637cfecac9941cecae: Status 404 returned error can't find the container with id 94d175a7a37c5c3ef8c10fc973b829f6b7a34e34bd1119637cfecac9941cecae
Sep 07 10:44:12.943614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:12.943530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" event={"ID":"958c82a1-332e-416d-9f03-7112419ca8f2","Type":"ContainerStarted","Data":"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"}
Sep 07 10:44:12.943614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:12.943594    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:12.943614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:12.943614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" event={"ID":"958c82a1-332e-416d-9f03-7112419ca8f2","Type":"ContainerStarted","Data":"94d175a7a37c5c3ef8c10fc973b829f6b7a34e34bd1119637cfecac9941cecae"}
Sep 07 10:44:12.962837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:12.962776    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" podStartSLOduration=1.9627617339999999 podStartE2EDuration="1.962761734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:12.962360204 +0000 UTC m=+1440.973013520" watchObservedRunningTime="2026-09-07 10:44:12.962761734 +0000 UTC m=+1440.973415043"
Sep 07 10:44:18.732429 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.732388    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:44:18.737491 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.737468    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.740348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.740324    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"openshift-service-ca.crt\""
Sep 07 10:44:18.741764 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.741743    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"default-dockercfg-8gzlr\""
Sep 07 10:44:18.742545 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.741772    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"auth-test-model-5a0ff8-kserve-self-signed-certs\""
Sep 07 10:44:18.742545 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.742068    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"kube-root-ca.crt\""
Sep 07 10:44:18.743974 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.743952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:44:18.884188 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.884393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.884393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.884393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.884569 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvd9\" (UniqueName: \"kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.884569 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.884472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985588 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvd9\" (UniqueName: \"kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985588 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.985816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.985707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.986071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.986052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.986131 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.986098    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.986131 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.986108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.988043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.988023    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.988200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.988184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:18.994707 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:18.994685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvd9\" (UniqueName: \"kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9\") pod \"auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:19.047998 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:19.047976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:19.340350 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:19.340296    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:44:19.342923 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:44:19.342895    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf336b59a_bd29_469f_8c8d_72e226ef1bbb.slice/crio-14f5b6301816f3ad9d77d7a07e50d6111f0645156e8dddff1c3b5b05649878a3 WatchSource:0}: Error finding container 14f5b6301816f3ad9d77d7a07e50d6111f0645156e8dddff1c3b5b05649878a3: Status 404 returned error can't find the container with id 14f5b6301816f3ad9d77d7a07e50d6111f0645156e8dddff1c3b5b05649878a3
Sep 07 10:44:19.974819 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:19.974781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" event={"ID":"f336b59a-bd29-469f-8c8d-72e226ef1bbb","Type":"ContainerStarted","Data":"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"}
Sep 07 10:44:19.975209 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:19.974830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" event={"ID":"f336b59a-bd29-469f-8c8d-72e226ef1bbb","Type":"ContainerStarted","Data":"14f5b6301816f3ad9d77d7a07e50d6111f0645156e8dddff1c3b5b05649878a3"}
Sep 07 10:44:19.994821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:19.994773    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" podStartSLOduration=1.9947599440000001 podStartE2EDuration="1.994759944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:19.993218984 +0000 UTC m=+1448.003872298" watchObservedRunningTime="2026-09-07 10:44:19.994759944 +0000 UTC m=+1448.005413294"
Sep 07 10:44:23.950689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:23.950654    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:44:26.005529 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.005494    2562 generic.go:356] "Generic (PLEG): container finished" podID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerID="70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6" exitCode=0
Sep 07 10:44:26.005873 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.005566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerDied","Data":"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"}
Sep 07 10:44:26.005916 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.005905    2562 scope.go:117] "RemoveContainer" containerID="70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"
Sep 07 10:44:26.006943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.006915    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:26.007031 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.007014    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:26.007073 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:26.007054    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:44:27.015012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:27.014947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerStarted","Data":"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"}
Sep 07 10:44:27.291032 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:27.290934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:27.291177 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:27.291129    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 10:44:27.291222 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:27.291177    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 10:44:28.291337 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:28.291279    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 10:44:28.291722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:28.291361    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 10:44:29.048559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:29.048532    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:29.048799 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:29.048776    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:29.060783 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:29.060760    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:29.290756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:29.290721    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 10:44:29.290927 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:29.290788    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 10:44:30.042927 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:30.042894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:44:30.291095 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:30.291063    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 10:44:30.291266 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:30.291124    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 10:44:31.291336 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:31.291282    2562 patch_prober.go:28] interesting pod/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 07 10:44:31.291714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:31.291362    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 07 10:44:32.417195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:32.417157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:32.417696 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:32.417573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:32.418292 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:32.418274    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:44:40.126714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.126679    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:44:40.130194 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.130179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.133081 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.133056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5e934f\""
Sep 07 10:44:40.133202 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.133089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5e934f-dockercfg-qhxfz\""
Sep 07 10:44:40.145994 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.145976    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:44:40.176119 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.176098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.176216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.176133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpsl\" (UniqueName: \"kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.277220 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.277195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpsl\" (UniqueName: \"kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.277385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.277333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.277996 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.277977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.286227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.286206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpsl\" (UniqueName: \"kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl\") pod \"payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.439942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.439879    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:40.708200 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:40.708178    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:44:40.710414 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:44:40.710384    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65419d0a_84a3_4288_a2b6_a6f49068669d.slice/crio-bd10ed59a05b3409b4cf65d0ad104349eb50c4f865c83f8e15084d2c1a6120c1 WatchSource:0}: Error finding container bd10ed59a05b3409b4cf65d0ad104349eb50c4f865c83f8e15084d2c1a6120c1: Status 404 returned error can't find the container with id bd10ed59a05b3409b4cf65d0ad104349eb50c4f865c83f8e15084d2c1a6120c1
Sep 07 10:44:41.078995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:41.078962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" event={"ID":"65419d0a-84a3-4288-a2b6-a6f49068669d","Type":"ContainerStarted","Data":"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"}
Sep 07 10:44:41.079169 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:41.079014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" event={"ID":"65419d0a-84a3-4288-a2b6-a6f49068669d","Type":"ContainerStarted","Data":"bd10ed59a05b3409b4cf65d0ad104349eb50c4f865c83f8e15084d2c1a6120c1"}
Sep 07 10:44:41.079169 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:41.079077    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:44:41.096627 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:41.096585    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" podStartSLOduration=1.096573083 podStartE2EDuration="1.096573083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:41.095218406 +0000 UTC m=+1469.105871720" watchObservedRunningTime="2026-09-07 10:44:41.096573083 +0000 UTC m=+1469.107226404"
Sep 07 10:44:52.084770 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:44:52.084740    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:45:00.149321 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.149276    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"]
Sep 07 10:45:00.153150 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.153133    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"]
Sep 07 10:45:00.153278 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.153256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"
Sep 07 10:45:00.156147 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.156119    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:45:00.156374 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.156289    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"
Sep 07 10:45:00.163060 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.163036    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"]
Sep 07 10:45:00.164912 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.164892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"]
Sep 07 10:45:00.167837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.167817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"
Sep 07 10:45:00.174228 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.174210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"
Sep 07 10:45:00.441867 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.441842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"]
Sep 07 10:45:00.444067 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:45:00.444034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3599b21c_2b0e_46b7_b850_49022fa943e6.slice/crio-95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271 WatchSource:0}: Error finding container 95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271: Status 404 returned error can't find the container with id 95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271
Sep 07 10:45:00.446693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:00.446671    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"]
Sep 07 10:45:00.450556 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:45:00.450533    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69044d71_a3fd_4331_ac7b_1bf1f15935f4.slice/crio-d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df WatchSource:0}: Error finding container d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df: Status 404 returned error can't find the container with id d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df
Sep 07 10:45:01.165245 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.165207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r" event={"ID":"69044d71-a3fd-4331-ac7b-1bf1f15935f4","Type":"ContainerStarted","Data":"42ecc7e97cec3cc7de72abd44cd2768b71c88d4fbd9c8b5ccfbb4f70ad4ec5f2"}
Sep 07 10:45:01.165702 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.165267    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r" event={"ID":"69044d71-a3fd-4331-ac7b-1bf1f15935f4","Type":"ContainerStarted","Data":"d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df"}
Sep 07 10:45:01.166697 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.166676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g" event={"ID":"3599b21c-2b0e-46b7-b850-49022fa943e6","Type":"ContainerStarted","Data":"210ab1e5c8b9a4cfb26e4d0e2d5c4f9fffa51a85c71be119740158e9adf25eb3"}
Sep 07 10:45:01.166814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.166711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g" event={"ID":"3599b21c-2b0e-46b7-b850-49022fa943e6","Type":"ContainerStarted","Data":"95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271"}
Sep 07 10:45:01.180850 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.180800    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r" podStartSLOduration=1.180787812 podStartE2EDuration="1.180787812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.178833755 +0000 UTC m=+1489.189487079" watchObservedRunningTime="2026-09-07 10:45:01.180787812 +0000 UTC m=+1489.191441126"
Sep 07 10:45:01.192848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:01.192811    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g" podStartSLOduration=1.192794109 podStartE2EDuration="1.192794109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.191984503 +0000 UTC m=+1489.202637816" watchObservedRunningTime="2026-09-07 10:45:01.192794109 +0000 UTC m=+1489.203447425"
Sep 07 10:45:02.172420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:02.172382    2562 generic.go:356] "Generic (PLEG): container finished" podID="69044d71-a3fd-4331-ac7b-1bf1f15935f4" containerID="42ecc7e97cec3cc7de72abd44cd2768b71c88d4fbd9c8b5ccfbb4f70ad4ec5f2" exitCode=0
Sep 07 10:45:02.172852 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:02.172472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r" event={"ID":"69044d71-a3fd-4331-ac7b-1bf1f15935f4","Type":"ContainerDied","Data":"42ecc7e97cec3cc7de72abd44cd2768b71c88d4fbd9c8b5ccfbb4f70ad4ec5f2"}
Sep 07 10:45:02.174078 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:02.174055    2562 generic.go:356] "Generic (PLEG): container finished" podID="3599b21c-2b0e-46b7-b850-49022fa943e6" containerID="210ab1e5c8b9a4cfb26e4d0e2d5c4f9fffa51a85c71be119740158e9adf25eb3" exitCode=0
Sep 07 10:45:02.174192 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:02.174117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g" event={"ID":"3599b21c-2b0e-46b7-b850-49022fa943e6","Type":"ContainerDied","Data":"210ab1e5c8b9a4cfb26e4d0e2d5c4f9fffa51a85c71be119740158e9adf25eb3"}
Sep 07 10:45:03.316714 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:03.316694    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"
Sep 07 10:45:03.319856 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:03.319836    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"
Sep 07 10:45:04.186065 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.186034    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"
Sep 07 10:45:04.186065 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.186065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r" event={"ID":"69044d71-a3fd-4331-ac7b-1bf1f15935f4","Type":"ContainerDied","Data":"d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df"}
Sep 07 10:45:04.186273 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.186107    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94427930c8c00dd54fa32558cc723ae057f1bd1ea15a6f4fb5b4753d5dca8df"
Sep 07 10:45:04.187395 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.187373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g" event={"ID":"3599b21c-2b0e-46b7-b850-49022fa943e6","Type":"ContainerDied","Data":"95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271"}
Sep 07 10:45:04.187525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.187398    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"
Sep 07 10:45:04.187525 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:04.187404    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95727850ffc519b26a8ae994b7f7e02c72a5f08060d52f3290033a6e8eb82271"
Sep 07 10:45:12.505564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:12.505539    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:45:12.507515 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:12.507493    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:45:12.580136 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:12.580116    2562 scope.go:117] "RemoveContainer" containerID="9d74ddeb2dbce4c7909bdb59650493b446e04be3c69d34b2df63900a1e62f5a4"
Sep 07 10:45:18.042107 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042072    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:45:18.042579 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042558    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3599b21c-2b0e-46b7-b850-49022fa943e6" containerName="cleanup"
Sep 07 10:45:18.042658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3599b21c-2b0e-46b7-b850-49022fa943e6" containerName="cleanup"
Sep 07 10:45:18.042658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042613    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69044d71-a3fd-4331-ac7b-1bf1f15935f4" containerName="cleanup"
Sep 07 10:45:18.042658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042622    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="69044d71-a3fd-4331-ac7b-1bf1f15935f4" containerName="cleanup"
Sep 07 10:45:18.042802 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042703    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3599b21c-2b0e-46b7-b850-49022fa943e6" containerName="cleanup"
Sep 07 10:45:18.042802 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.042716    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="69044d71-a3fd-4331-ac7b-1bf1f15935f4" containerName="cleanup"
Sep 07 10:45:18.047092 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.047072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.050052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.050027    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2700585f-openshift-default-dockercfg-rdr4c\""
Sep 07 10:45:18.058220 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.058197    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:45:18.179196 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179196 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179248    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvf2l\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.179692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.179521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280577 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvf2l\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.280766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.281163 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.280990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.281163 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.281090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.281163 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.281126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.281364 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.281175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.281531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.281456    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.283030 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.283007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.283149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.283131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.290522 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.290505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.290689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.290660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvf2l\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l\") pod \"e2e-mt-2700585f-openshift-default-855d469f9b-n88d9\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") " pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.357931 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.357902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:18.633802 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.633728    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:45:18.636056 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:45:18.636031    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9aeea61_a036_4fbc_bb93_90c7d992ccc4.slice/crio-376e620719261d761406a375ed9093a4aba9d86b0ee75b2c76cee1a3e113a877 WatchSource:0}: Error finding container 376e620719261d761406a375ed9093a4aba9d86b0ee75b2c76cee1a3e113a877: Status 404 returned error can't find the container with id 376e620719261d761406a375ed9093a4aba9d86b0ee75b2c76cee1a3e113a877
Sep 07 10:45:18.638204 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.638172    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:18.638324 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.638233    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:18.638324 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:18.638260    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:19.253927 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:19.253897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerStarted","Data":"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"}
Sep 07 10:45:19.254297 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:19.253943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerStarted","Data":"376e620719261d761406a375ed9093a4aba9d86b0ee75b2c76cee1a3e113a877"}
Sep 07 10:45:19.275051 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:19.275006    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podStartSLOduration=1.274993196 podStartE2EDuration="1.274993196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:19.273891871 +0000 UTC m=+1507.284545187" watchObservedRunningTime="2026-09-07 10:45:19.274993196 +0000 UTC m=+1507.285646551"
Sep 07 10:45:19.358775 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:19.358746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:19.363634 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:19.363615    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:20.258468 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:20.258435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:20.259370 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:20.259352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:42.365418 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.365388    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerID="9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa" exitCode=0
Sep 07 10:45:42.365869 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.365469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerDied","Data":"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"}
Sep 07 10:45:42.365869 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.365815    2562 scope.go:117] "RemoveContainer" containerID="9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"
Sep 07 10:45:42.366806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.366781    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:42.366881 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.366868    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:42.366925 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:42.366906    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:45:43.371756 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:43.371722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerStarted","Data":"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"}
Sep 07 10:45:44.358436 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:44.358394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:44.358689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:44.358666    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:44.358796 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:44.358737    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:45.359131 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:45.359098    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:45.359534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:45.359154    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:46.358832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:46.358792    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:46.359018 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:46.358852    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:47.359037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:47.359002    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:47.359441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:47.359055    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:48.359129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:48.359095    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:48.359537 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:48.359147    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:49.359059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:49.359022    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 07 10:45:49.359265 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:49.359097    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 07 10:45:50.258373 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:50.258340    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:50.362317 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:50.362281    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:45:50.406710 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:45:50.406690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:46:02.734030 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.733997    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:46:02.736251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.736235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.739073 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.739054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2700585f\"/\"openshift-service-ca.crt\""
Sep 07 10:46:02.740348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.740325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2700585f\"/\"default-dockercfg-l48b2\""
Sep 07 10:46:02.740348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.740343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2700585f\"/\"kube-root-ca.crt\""
Sep 07 10:46:02.740534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.740386    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2700585f\"/\"e2e-lifecycle-model-2700585f-kserve-self-signed-certs\""
Sep 07 10:46:02.746572 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.746549    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:46:02.857316 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.857473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.857473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xk5\" (UniqueName: \"kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.857473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.857473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.857473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.857454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958295 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958295 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xk5\" (UniqueName: \"kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958531 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958750 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958728    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.958825 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.958810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.960754 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.960736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.960924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.960908    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:02.967605 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:02.967581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xk5\" (UniqueName: \"kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5\") pod \"e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") " pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:03.046516 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:03.046451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:03.320328 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:03.320288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:46:03.322010 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:46:03.321986    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76e27480_8412_475d_af0b_9beb5eb08413.slice/crio-471865f6035d61255eb63635ebe273bc398e73dc42ade7a2d8a75f4301b11b26 WatchSource:0}: Error finding container 471865f6035d61255eb63635ebe273bc398e73dc42ade7a2d8a75f4301b11b26: Status 404 returned error can't find the container with id 471865f6035d61255eb63635ebe273bc398e73dc42ade7a2d8a75f4301b11b26
Sep 07 10:46:03.463864 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:03.463824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" event={"ID":"76e27480-8412-475d-af0b-9beb5eb08413","Type":"ContainerStarted","Data":"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"}
Sep 07 10:46:03.463864 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:03.463867    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" event={"ID":"76e27480-8412-475d-af0b-9beb5eb08413","Type":"ContainerStarted","Data":"471865f6035d61255eb63635ebe273bc398e73dc42ade7a2d8a75f4301b11b26"}
Sep 07 10:46:03.482135 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:03.482088    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" podStartSLOduration=1.48207356 podStartE2EDuration="1.48207356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:46:03.481142827 +0000 UTC m=+1551.491796141" watchObservedRunningTime="2026-09-07 10:46:03.48207356 +0000 UTC m=+1551.492726873"
Sep 07 10:46:13.046690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:13.046636    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:13.046690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:13.046698    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:13.059018 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:13.058990    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:46:13.524218 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:46:13.524189    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:47:07.117644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:07.117604    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:47:22.137813 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.137752    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" podUID="76e27480-8412-475d-af0b-9beb5eb08413" containerName="main" containerID="cri-o://577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41" gracePeriod=45
Sep 07 10:47:22.386435 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.386409    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:47:22.487513 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487439    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6xk5\" (UniqueName: \"kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487513 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487477    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487720    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm\") pod \"76e27480-8412-475d-af0b-9beb5eb08413\" (UID: \"76e27480-8412-475d-af0b-9beb5eb08413\") "
Sep 07 10:47:22.487926 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487746    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache" (OuterVolumeSpecName: "model-cache") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:22.487983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:22.487983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.487941    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home" (OuterVolumeSpecName: "home") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:22.488086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.488052    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.488086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.488070    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.488086 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.488083    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.489846 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.489811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5" (OuterVolumeSpecName: "kube-api-access-v6xk5") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "kube-api-access-v6xk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:22.489967 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.489842    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:22.490012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.489954    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm" (OuterVolumeSpecName: "dshm") pod "76e27480-8412-475d-af0b-9beb5eb08413" (UID: "76e27480-8412-475d-af0b-9beb5eb08413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:22.589440 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.589404    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6xk5\" (UniqueName: \"kubernetes.io/projected/76e27480-8412-475d-af0b-9beb5eb08413-kube-api-access-v6xk5\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.589440 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.589435    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76e27480-8412-475d-af0b-9beb5eb08413-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.589440 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.589449    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76e27480-8412-475d-af0b-9beb5eb08413-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:22.837076 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.837038    2562 generic.go:356] "Generic (PLEG): container finished" podID="76e27480-8412-475d-af0b-9beb5eb08413" containerID="577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41" exitCode=0
Sep 07 10:47:22.837076 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.837068    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"
Sep 07 10:47:22.837372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.837101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" event={"ID":"76e27480-8412-475d-af0b-9beb5eb08413","Type":"ContainerDied","Data":"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"}
Sep 07 10:47:22.837372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.837149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w" event={"ID":"76e27480-8412-475d-af0b-9beb5eb08413","Type":"ContainerDied","Data":"471865f6035d61255eb63635ebe273bc398e73dc42ade7a2d8a75f4301b11b26"}
Sep 07 10:47:22.837372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.837167    2562 scope.go:117] "RemoveContainer" containerID="577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"
Sep 07 10:47:22.846316 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.846283    2562 scope.go:117] "RemoveContainer" containerID="577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"
Sep 07 10:47:22.846614 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:47:22.846595    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41\": container with ID starting with 577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41 not found: ID does not exist" containerID="577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"
Sep 07 10:47:22.846695 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.846621    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41"} err="failed to get container status \"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41\": rpc error: code = NotFound desc = could not find container \"577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41\": container with ID starting with 577bdc0c5fead367c731139cd928cc94f413d47de2b565bf9cb7d2788777ac41 not found: ID does not exist"
Sep 07 10:47:22.854481 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.854461    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:47:22.863361 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:22.863343    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2700585f/e2e-lifecycle-model-2700585f-kserve-7754d58c5-c242w"]
Sep 07 10:47:24.496279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:24.496246    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e27480-8412-475d-af0b-9beb5eb08413" path="/var/lib/kubelet/pods/76e27480-8412-475d-af0b-9beb5eb08413/volumes"
Sep 07 10:47:50.357905 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:50.357871    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:47:50.358476 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:50.358096    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" containerID="cri-o://d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c" gracePeriod=30
Sep 07 10:47:50.408131 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:50.408094    2562 patch_prober.go:28] interesting pod/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 10:47:50.408260 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:50.408152    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 10:47:55.608961 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.608934    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:47:55.691965 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.691867    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.691965 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.691924    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.691965 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.691949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.691985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvf2l\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692038    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692110    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692225 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo\") pod \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\" (UID: \"f9aeea61-a036-4fbc-bb93-90c7d992ccc4\") "
Sep 07 10:47:55.692599 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692395    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:55.692599 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692468    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:47:55.692599 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:55.692599 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:55.692774 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.692593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:55.694347 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.694269    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:55.694459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.694427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 10:47:55.694557 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.694533    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l" (OuterVolumeSpecName: "kube-api-access-cvf2l") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "kube-api-access-cvf2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:55.694601 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.694554    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9aeea61-a036-4fbc-bb93-90c7d992ccc4" (UID: "f9aeea61-a036-4fbc-bb93-90c7d992ccc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:55.793142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793102    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvf2l\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-kube-api-access-cvf2l\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793136    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793142 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793145    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-credential-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793154    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-data\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793162    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-envoy\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793172    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-podinfo\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793179    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istiod-ca-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793186    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-istio-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.793420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.793194    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9aeea61-a036-4fbc-bb93-90c7d992ccc4-workload-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:47:55.994338 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.994220    2562 generic.go:356] "Generic (PLEG): container finished" podID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerID="d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c" exitCode=0
Sep 07 10:47:55.994338 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.994266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"
Sep 07 10:47:55.994640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.994338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerDied","Data":"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"}
Sep 07 10:47:55.994640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.994397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9" event={"ID":"f9aeea61-a036-4fbc-bb93-90c7d992ccc4","Type":"ContainerDied","Data":"376e620719261d761406a375ed9093a4aba9d86b0ee75b2c76cee1a3e113a877"}
Sep 07 10:47:55.994640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:55.994422    2562 scope.go:117] "RemoveContainer" containerID="d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"
Sep 07 10:47:56.003605 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.003585    2562 scope.go:117] "RemoveContainer" containerID="9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"
Sep 07 10:47:56.011747 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.011727    2562 scope.go:117] "RemoveContainer" containerID="d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"
Sep 07 10:47:56.012071 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:47:56.012051    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c\": container with ID starting with d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c not found: ID does not exist" containerID="d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"
Sep 07 10:47:56.012116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.012081    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c"} err="failed to get container status \"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c\": rpc error: code = NotFound desc = could not find container \"d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c\": container with ID starting with d5161c2047935b4833faca04bfae2606465bc721ed404bf33d5b4b1c8044516c not found: ID does not exist"
Sep 07 10:47:56.012116 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.012101    2562 scope.go:117] "RemoveContainer" containerID="9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"
Sep 07 10:47:56.012359 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:47:56.012341    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa\": container with ID starting with 9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa not found: ID does not exist" containerID="9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"
Sep 07 10:47:56.012426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.012369    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa"} err="failed to get container status \"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa\": rpc error: code = NotFound desc = could not find container \"9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa\": container with ID starting with 9a6940d90cb4b8dccc8d6fb2295ea45ffdca27e2279119b03c8f536d3c943aaa not found: ID does not exist"
Sep 07 10:47:56.017039 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.017010    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:47:56.020622 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.020600    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2700585f-openshift-default-855d469f9b-n88d9"]
Sep 07 10:47:56.484656 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.484626    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:47:56.495918 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:47:56.495877    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" path="/var/lib/kubelet/pods/f9aeea61-a036-4fbc-bb93-90c7d992ccc4/volumes"
Sep 07 10:48:07.758419 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758377    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758736    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758746    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758752    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758772    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e27480-8412-475d-af0b-9beb5eb08413" containerName="main"
Sep 07 10:48:07.758811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758777    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e27480-8412-475d-af0b-9beb5eb08413" containerName="main"
Sep 07 10:48:07.759007 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758828    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e27480-8412-475d-af0b-9beb5eb08413" containerName="main"
Sep 07 10:48:07.759007 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758837    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.759007 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.758843    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9aeea61-a036-4fbc-bb93-90c7d992ccc4" containerName="istio-proxy"
Sep 07 10:48:07.763178 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.763161    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.768272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.768242    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"sub-test-model-5a0ff8-kserve-self-signed-certs\""
Sep 07 10:48:07.776988 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.776960    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:48:07.902365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtx96\" (UniqueName: \"kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.902570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.902570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.902570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.902570 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:07.902771 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:07.902632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004013 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.003977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtx96\" (UniqueName: \"kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004227 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.004666 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.004607    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.006658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.006639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.006813 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.006795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.012002 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.011945    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtx96\" (UniqueName: \"kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96\") pod \"sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.072418 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.072379    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:08.344834 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:08.344748    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:48:08.347089 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:48:08.347068    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7030d21_064e_48d9_abe9_41e6101d2dd5.slice/crio-d54f4fd732cf1f04b9f00a3642b57a44cee5a74575e66668caf490229d2d228d WatchSource:0}: Error finding container d54f4fd732cf1f04b9f00a3642b57a44cee5a74575e66668caf490229d2d228d: Status 404 returned error can't find the container with id d54f4fd732cf1f04b9f00a3642b57a44cee5a74575e66668caf490229d2d228d
Sep 07 10:48:09.057029 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:09.056998    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" event={"ID":"b7030d21-064e-48d9-abe9-41e6101d2dd5","Type":"ContainerStarted","Data":"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"}
Sep 07 10:48:09.057029 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:09.057036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" event={"ID":"b7030d21-064e-48d9-abe9-41e6101d2dd5","Type":"ContainerStarted","Data":"d54f4fd732cf1f04b9f00a3642b57a44cee5a74575e66668caf490229d2d228d"}
Sep 07 10:48:09.077181 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:09.077126    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" podStartSLOduration=2.077109137 podStartE2EDuration="2.077109137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:09.074493921 +0000 UTC m=+1677.085147227" watchObservedRunningTime="2026-09-07 10:48:09.077109137 +0000 UTC m=+1677.087762469"
Sep 07 10:48:11.505523 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.505452    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" podUID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" containerName="main" containerID="cri-o://1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f" gracePeriod=45
Sep 07 10:48:11.758795 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.758725    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:48:11.837962 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.837926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.837972    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngvd9\" (UniqueName: \"kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838002    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838053    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838098    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838148 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm\") pod \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\" (UID: \"f336b59a-bd29-469f-8c8d-72e226ef1bbb\") "
Sep 07 10:48:11.838456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home" (OuterVolumeSpecName: "home") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:48:11.838456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838355    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:48:11.838456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838391    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:48:11.838456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838441    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:11.838456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838457    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:11.838668 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.838467    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:11.840479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.840443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm" (OuterVolumeSpecName: "dshm") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:48:11.840607 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.840523    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:48:11.840607 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.840586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9" (OuterVolumeSpecName: "kube-api-access-ngvd9") pod "f336b59a-bd29-469f-8c8d-72e226ef1bbb" (UID: "f336b59a-bd29-469f-8c8d-72e226ef1bbb"). InnerVolumeSpecName "kube-api-access-ngvd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:48:11.939598 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.939561    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f336b59a-bd29-469f-8c8d-72e226ef1bbb-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:11.939598 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.939592    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvd9\" (UniqueName: \"kubernetes.io/projected/f336b59a-bd29-469f-8c8d-72e226ef1bbb-kube-api-access-ngvd9\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:11.939598 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:11.939605    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f336b59a-bd29-469f-8c8d-72e226ef1bbb-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:48:12.072702 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.072668    2562 generic.go:356] "Generic (PLEG): container finished" podID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" containerID="1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f" exitCode=0
Sep 07 10:48:12.072907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.072715    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"
Sep 07 10:48:12.072907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.072755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" event={"ID":"f336b59a-bd29-469f-8c8d-72e226ef1bbb","Type":"ContainerDied","Data":"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"}
Sep 07 10:48:12.072907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.072802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2" event={"ID":"f336b59a-bd29-469f-8c8d-72e226ef1bbb","Type":"ContainerDied","Data":"14f5b6301816f3ad9d77d7a07e50d6111f0645156e8dddff1c3b5b05649878a3"}
Sep 07 10:48:12.072907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.072826    2562 scope.go:117] "RemoveContainer" containerID="1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"
Sep 07 10:48:12.082253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.082234    2562 scope.go:117] "RemoveContainer" containerID="1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"
Sep 07 10:48:12.082599 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:48:12.082579    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f\": container with ID starting with 1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f not found: ID does not exist" containerID="1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"
Sep 07 10:48:12.082677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.082612    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f"} err="failed to get container status \"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f\": rpc error: code = NotFound desc = could not find container \"1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f\": container with ID starting with 1cf1683ea367bbe518a5057e101376b31a5fcbe63521aed1554527a95ef53f0f not found: ID does not exist"
Sep 07 10:48:12.094692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.094665    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:48:12.099326 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.099275    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/auth-test-model-5a0ff8-kserve-587d8f5bc-qjdt2"]
Sep 07 10:48:12.496278 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.496198    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" path="/var/lib/kubelet/pods/f336b59a-bd29-469f-8c8d-72e226ef1bbb/volumes"
Sep 07 10:48:12.610604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:12.610571    2562 scope.go:117] "RemoveContainer" containerID="b5ab57ba24084cbe07f2038809188df8ad85844cf50b0087e462d86f792b212c"
Sep 07 10:48:18.072768 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:18.072716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:18.073190 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:18.072794    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:18.085767 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:18.085734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:18.111171 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:18.111132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:48:25.515973 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.515930    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:48:25.516630 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.516455    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" containerName="main"
Sep 07 10:48:25.516630 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.516475    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" containerName="main"
Sep 07 10:48:25.516630 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.516605    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f336b59a-bd29-469f-8c8d-72e226ef1bbb" containerName="main"
Sep 07 10:48:25.519712 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.519692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.522496 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.522473    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"openshift-service-ca.crt\""
Sep 07 10:48:25.523537 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.523512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"sub-test-model-659410-kserve-self-signed-certs\""
Sep 07 10:48:25.523681 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.523562    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"default-dockercfg-8q89w\""
Sep 07 10:48:25.523681 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.523638    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"kube-root-ca.crt\""
Sep 07 10:48:25.534940 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.534912    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:48:25.565640 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.565814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.565814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.565814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qqk\" (UniqueName: \"kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.565814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.565814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.565803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666486 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qqk\" (UniqueName: \"kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666693 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666824 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666928 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.666928 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.667050 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.666965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.667170 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.667152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.669174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.669153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.669284 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.669249    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.674905 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.674881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qqk\" (UniqueName: \"kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk\") pod \"sub-test-model-659410-kserve-674b6d474f-4t6qc\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") " pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:25.836786 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:25.836753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:26.114327 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:26.114281    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:48:26.116521 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:48:26.116492    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5fc48fc_bdc0_44e8_8c49_be3b45091de7.slice/crio-3be2053726de9e859859faa3d5f7b8722ef53cbe103ecaa2504af3261a0dca2b WatchSource:0}: Error finding container 3be2053726de9e859859faa3d5f7b8722ef53cbe103ecaa2504af3261a0dca2b: Status 404 returned error can't find the container with id 3be2053726de9e859859faa3d5f7b8722ef53cbe103ecaa2504af3261a0dca2b
Sep 07 10:48:26.135262 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:26.135232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" event={"ID":"e5fc48fc-bdc0-44e8-8c49-be3b45091de7","Type":"ContainerStarted","Data":"3be2053726de9e859859faa3d5f7b8722ef53cbe103ecaa2504af3261a0dca2b"}
Sep 07 10:48:27.141772 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:27.141609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" event={"ID":"e5fc48fc-bdc0-44e8-8c49-be3b45091de7","Type":"ContainerStarted","Data":"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"}
Sep 07 10:48:27.159237 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:27.159186    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" podStartSLOduration=2.159164976 podStartE2EDuration="2.159164976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:27.158188504 +0000 UTC m=+1695.168841819" watchObservedRunningTime="2026-09-07 10:48:27.159164976 +0000 UTC m=+1695.169818292"
Sep 07 10:48:35.837369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:35.837270    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:35.837369 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:35.837353    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:35.850321 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:35.850269    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:36.199064 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:36.198970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:48:50.600757 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.600721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:48:50.604099 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.604076    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.606507 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.606485    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4e2c0-dockercfg-lz4tj\""
Sep 07 10:48:50.606617 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.606485    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4e2c0\""
Sep 07 10:48:50.613722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.613696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:48:50.700378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.700338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.700556 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.700424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzgb\" (UniqueName: \"kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.801656 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.801619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzgb\" (UniqueName: \"kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.801874 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.801694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.802412 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.802393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.809849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.809831    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzgb\" (UniqueName: \"kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb\") pod \"payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:50.915296 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:50.915206    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:51.193353 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:51.193329    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:48:51.195365 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:48:51.195339    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655e96af_ef99_4d75_9d40_78931bdce5e9.slice/crio-09347ba4e26279884a5b73b0be38b285e03324bef8ed28f7b393c3e21f800102 WatchSource:0}: Error finding container 09347ba4e26279884a5b73b0be38b285e03324bef8ed28f7b393c3e21f800102: Status 404 returned error can't find the container with id 09347ba4e26279884a5b73b0be38b285e03324bef8ed28f7b393c3e21f800102
Sep 07 10:48:51.254597 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:51.254569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" event={"ID":"655e96af-ef99-4d75-9d40-78931bdce5e9","Type":"ContainerStarted","Data":"09347ba4e26279884a5b73b0be38b285e03324bef8ed28f7b393c3e21f800102"}
Sep 07 10:48:52.260481 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:52.260440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" event={"ID":"655e96af-ef99-4d75-9d40-78931bdce5e9","Type":"ContainerStarted","Data":"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"}
Sep 07 10:48:52.260900 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:52.260561    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:48:52.277849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:48:52.277793    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" podStartSLOduration=2.277778217 podStartE2EDuration="2.277778217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:52.276522687 +0000 UTC m=+1720.287176003" watchObservedRunningTime="2026-09-07 10:48:52.277778217 +0000 UTC m=+1720.288431546"
Sep 07 10:49:03.267202 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:03.267168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:49:19.833026 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.832983    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:49:19.836507 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.836483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:19.839058 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.839031    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4f3dee\""
Sep 07 10:49:19.839253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.839233    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4f3dee-dockercfg-8hzxl\""
Sep 07 10:49:19.848536 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.848510    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:49:19.942262 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.942229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:19.942440 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:19.942286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbfzl\" (UniqueName: \"kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.043088 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.043053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbfzl\" (UniqueName: \"kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.043256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.043149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.043849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.043828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.051413 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.051389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbfzl\" (UniqueName: \"kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl\") pod \"payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.147167 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.147088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:20.418997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:20.418970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:49:20.421910 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:49:20.421877    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21af99b3_de35_47ba_8ca2_53238968bc08.slice/crio-b639c89ac4791e68ef7e6ff8c72b0bd784ca22b8065b4c0f22970513fcc81b6a WatchSource:0}: Error finding container b639c89ac4791e68ef7e6ff8c72b0bd784ca22b8065b4c0f22970513fcc81b6a: Status 404 returned error can't find the container with id b639c89ac4791e68ef7e6ff8c72b0bd784ca22b8065b4c0f22970513fcc81b6a
Sep 07 10:49:21.403083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:21.403042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" event={"ID":"21af99b3-de35-47ba-8ca2-53238968bc08","Type":"ContainerStarted","Data":"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"}
Sep 07 10:49:21.403505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:21.403106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" event={"ID":"21af99b3-de35-47ba-8ca2-53238968bc08","Type":"ContainerStarted","Data":"b639c89ac4791e68ef7e6ff8c72b0bd784ca22b8065b4c0f22970513fcc81b6a"}
Sep 07 10:49:21.403505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:21.403168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:21.434574 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:21.434513    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" podStartSLOduration=2.43449689 podStartE2EDuration="2.43449689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:49:21.432090525 +0000 UTC m=+1749.442743840" watchObservedRunningTime="2026-09-07 10:49:21.43449689 +0000 UTC m=+1749.445150205"
Sep 07 10:49:30.724442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:30.724409    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:49:32.410228 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:32.410196    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:49:32.936563 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:32.936529    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:49:37.532520 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.532488    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:49:37.542315 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.542272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.544944 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.544918    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:49:37.545139 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.545124    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5a0ff8\"/\"rate-test-model-5a0ff8-kserve-self-signed-certs\""
Sep 07 10:49:37.704942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.704905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.704942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.704950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr877\" (UniqueName: \"kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.705187 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.704983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.705187 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.705009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.705187 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.705055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.705187 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.705092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr877\" (UniqueName: \"kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806403 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806403 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806506 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806506 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.806610 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.806543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.808811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.808790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.808991 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.808975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.817443 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.817417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr877\" (UniqueName: \"kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877\") pod \"rate-test-model-5a0ff8-kserve-6557df9f-qxqv7\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") " pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:37.854587 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:37.854554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:38.130042 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:38.130014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:49:38.132229 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:49:38.132202    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe609053_da73_4ab6_8a61_7649a4f5bb08.slice/crio-ea30da70d0e972952f911a690292e46f078978ba045d6b8975a7d7a35180ed06 WatchSource:0}: Error finding container ea30da70d0e972952f911a690292e46f078978ba045d6b8975a7d7a35180ed06: Status 404 returned error can't find the container with id ea30da70d0e972952f911a690292e46f078978ba045d6b8975a7d7a35180ed06
Sep 07 10:49:38.483323 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:38.483219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" event={"ID":"fe609053-da73-4ab6-8a61-7649a4f5bb08","Type":"ContainerStarted","Data":"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"}
Sep 07 10:49:38.483323 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:38.483260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" event={"ID":"fe609053-da73-4ab6-8a61-7649a4f5bb08","Type":"ContainerStarted","Data":"ea30da70d0e972952f911a690292e46f078978ba045d6b8975a7d7a35180ed06"}
Sep 07 10:49:38.502665 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:38.502612    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" podStartSLOduration=1.502596743 podStartE2EDuration="1.502596743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:49:38.501164456 +0000 UTC m=+1766.511817783" watchObservedRunningTime="2026-09-07 10:49:38.502596743 +0000 UTC m=+1766.513250057"
Sep 07 10:49:45.744477 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.744413    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" podUID="b7030d21-064e-48d9-abe9-41e6101d2dd5" containerName="main" containerID="cri-o://8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7" gracePeriod=45
Sep 07 10:49:45.980442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.980421    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:49:45.987118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987101    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987180 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987180 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987157    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtx96\" (UniqueName: \"kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987366 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987272    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir\") pod \"b7030d21-064e-48d9-abe9-41e6101d2dd5\" (UID: \"b7030d21-064e-48d9-abe9-41e6101d2dd5\") "
Sep 07 10:49:45.987453 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987394    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:45.987537 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987473    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home" (OuterVolumeSpecName: "home") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:45.987600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987561    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:45.987600 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987576    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:45.987689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.987615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:45.989367 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.989345    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm" (OuterVolumeSpecName: "dshm") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:45.989464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.989445    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:49:45.989508 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:45.989451    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96" (OuterVolumeSpecName: "kube-api-access-xtx96") pod "b7030d21-064e-48d9-abe9-41e6101d2dd5" (UID: "b7030d21-064e-48d9-abe9-41e6101d2dd5"). InnerVolumeSpecName "kube-api-access-xtx96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:49:46.088774 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.088746    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtx96\" (UniqueName: \"kubernetes.io/projected/b7030d21-064e-48d9-abe9-41e6101d2dd5-kube-api-access-xtx96\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:46.088774 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.088776    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:46.088977 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.088786    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7030d21-064e-48d9-abe9-41e6101d2dd5-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:46.088977 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.088794    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7030d21-064e-48d9-abe9-41e6101d2dd5-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:46.522563 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.522532    2562 generic.go:356] "Generic (PLEG): container finished" podID="b7030d21-064e-48d9-abe9-41e6101d2dd5" containerID="8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7" exitCode=0
Sep 07 10:49:46.522737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.522574    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"
Sep 07 10:49:46.522737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.522620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" event={"ID":"b7030d21-064e-48d9-abe9-41e6101d2dd5","Type":"ContainerDied","Data":"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"}
Sep 07 10:49:46.522737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.522653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h" event={"ID":"b7030d21-064e-48d9-abe9-41e6101d2dd5","Type":"ContainerDied","Data":"d54f4fd732cf1f04b9f00a3642b57a44cee5a74575e66668caf490229d2d228d"}
Sep 07 10:49:46.522737 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.522671    2562 scope.go:117] "RemoveContainer" containerID="8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"
Sep 07 10:49:46.531412 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.531395    2562 scope.go:117] "RemoveContainer" containerID="8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"
Sep 07 10:49:46.531710 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:49:46.531691    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7\": container with ID starting with 8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7 not found: ID does not exist" containerID="8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"
Sep 07 10:49:46.531757 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.531718    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7"} err="failed to get container status \"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7\": rpc error: code = NotFound desc = could not find container \"8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7\": container with ID starting with 8d9ba75c8ef9906147e4af38521981fc9b5a6d0d41d0d1f2548a603dac9fb9f7 not found: ID does not exist"
Sep 07 10:49:46.539792 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.539765    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:49:46.541443 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:46.541425    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/sub-test-model-5a0ff8-kserve-85bd6fc675-29v7h"]
Sep 07 10:49:47.104002 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.103967    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"]
Sep 07 10:49:47.104397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.104323    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7030d21-064e-48d9-abe9-41e6101d2dd5" containerName="main"
Sep 07 10:49:47.104397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.104334    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7030d21-064e-48d9-abe9-41e6101d2dd5" containerName="main"
Sep 07 10:49:47.104397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.104396    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7030d21-064e-48d9-abe9-41e6101d2dd5" containerName="main"
Sep 07 10:49:47.108782 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.108751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:47.111959 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.111936    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:49:47.117760 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.117723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"]
Sep 07 10:49:47.198224 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.198180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt\" (UID: \"b4f29d70-f3f3-4ced-b56d-9535ba84092c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:47.298809 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.298755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt\" (UID: \"b4f29d70-f3f3-4ced-b56d-9535ba84092c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:47.299555 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.299533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt\" (UID: \"b4f29d70-f3f3-4ced-b56d-9535ba84092c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:47.422464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.422385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:47.695814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.695791    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"]
Sep 07 10:49:47.698416 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:49:47.698379    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f29d70_f3f3_4ced_b56d_9535ba84092c.slice/crio-48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f WatchSource:0}: Error finding container 48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f: Status 404 returned error can't find the container with id 48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f
Sep 07 10:49:47.855712 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.855669    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:47.855909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.855731    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:47.873016 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.872983    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:47.956382 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:47.956277    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" podUID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" containerName="main" containerID="cri-o://cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635" gracePeriod=45
Sep 07 10:49:48.193907 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.193877    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:49:48.204466 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204448    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204548 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204484    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204548 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204617 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204617 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204582    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204709 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8qqk\" (UniqueName: \"kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk\") pod \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\" (UID: \"e5fc48fc-bdc0-44e8-8c49-be3b45091de7\") "
Sep 07 10:49:48.204861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204742    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:48.204861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home" (OuterVolumeSpecName: "home") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:48.204861 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:48.205045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204937    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.205045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204954    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.205045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.204962    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.206832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.206784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm" (OuterVolumeSpecName: "dshm") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:49:48.206832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.206821    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:49:48.206941 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.206875    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk" (OuterVolumeSpecName: "kube-api-access-l8qqk") pod "e5fc48fc-bdc0-44e8-8c49-be3b45091de7" (UID: "e5fc48fc-bdc0-44e8-8c49-be3b45091de7"). InnerVolumeSpecName "kube-api-access-l8qqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:49:48.305979 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.305933    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8qqk\" (UniqueName: \"kubernetes.io/projected/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-kube-api-access-l8qqk\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.305979 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.305964    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.305979 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.305974    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fc48fc-bdc0-44e8-8c49-be3b45091de7-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:48.496645 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.496562    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7030d21-064e-48d9-abe9-41e6101d2dd5" path="/var/lib/kubelet/pods/b7030d21-064e-48d9-abe9-41e6101d2dd5/volumes"
Sep 07 10:49:48.534841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.534813    2562 generic.go:356] "Generic (PLEG): container finished" podID="b4f29d70-f3f3-4ced-b56d-9535ba84092c" containerID="f59efa9fe08baaf47adb8b03a105d5ae1214bb741b58dd131a79b431cada5801" exitCode=0
Sep 07 10:49:48.534978 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.534899    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt" event={"ID":"b4f29d70-f3f3-4ced-b56d-9535ba84092c","Type":"ContainerDied","Data":"f59efa9fe08baaf47adb8b03a105d5ae1214bb741b58dd131a79b431cada5801"}
Sep 07 10:49:48.534978 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.534945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt" event={"ID":"b4f29d70-f3f3-4ced-b56d-9535ba84092c","Type":"ContainerStarted","Data":"48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f"}
Sep 07 10:49:48.536251 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.536227    2562 generic.go:356] "Generic (PLEG): container finished" podID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" containerID="cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635" exitCode=0
Sep 07 10:49:48.536362 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.536261    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"
Sep 07 10:49:48.536362 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.536294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" event={"ID":"e5fc48fc-bdc0-44e8-8c49-be3b45091de7","Type":"ContainerDied","Data":"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"}
Sep 07 10:49:48.536362 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.536355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc" event={"ID":"e5fc48fc-bdc0-44e8-8c49-be3b45091de7","Type":"ContainerDied","Data":"3be2053726de9e859859faa3d5f7b8722ef53cbe103ecaa2504af3261a0dca2b"}
Sep 07 10:49:48.536479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.536377    2562 scope.go:117] "RemoveContainer" containerID="cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"
Sep 07 10:49:48.546181 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.546158    2562 scope.go:117] "RemoveContainer" containerID="cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"
Sep 07 10:49:48.546510 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:49:48.546485    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635\": container with ID starting with cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635 not found: ID does not exist" containerID="cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"
Sep 07 10:49:48.546614 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.546516    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635"} err="failed to get container status \"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635\": rpc error: code = NotFound desc = could not find container \"cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635\": container with ID starting with cd325b1c601ad15b2b09225c4fb204a0588597f5c2f20cfc66d22e9b82460635 not found: ID does not exist"
Sep 07 10:49:48.551159 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.551130    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:49:48.563314 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.563273    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:49:48.567722 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:48.567700    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/sub-test-model-659410-kserve-674b6d474f-4t6qc"]
Sep 07 10:49:49.667576 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:49.667555    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:49:49.719209 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:49.719184    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca\") pod \"b4f29d70-f3f3-4ced-b56d-9535ba84092c\" (UID: \"b4f29d70-f3f3-4ced-b56d-9535ba84092c\") "
Sep 07 10:49:49.719616 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:49.719596    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4f29d70-f3f3-4ced-b56d-9535ba84092c" (UID: "b4f29d70-f3f3-4ced-b56d-9535ba84092c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:49:49.820294 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:49.820267    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f29d70-f3f3-4ced-b56d-9535ba84092c-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:49:50.495823 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:50.495790    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" path="/var/lib/kubelet/pods/e5fc48fc-bdc0-44e8-8c49-be3b45091de7/volumes"
Sep 07 10:49:50.547426 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:50.547383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt" event={"ID":"b4f29d70-f3f3-4ced-b56d-9535ba84092c","Type":"ContainerDied","Data":"48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f"}
Sep 07 10:49:50.547641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:50.547437    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48cc448399fb751d6f4dcd1eecfe69f54e5fa1bd9b4d94a35f9f101df8c70e9f"
Sep 07 10:49:50.547641 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:49:50.547402    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"
Sep 07 10:50:02.710732 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:02.710694    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:50:02.711292 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:02.710903    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" podUID="655e96af-ef99-4d75-9d40-78931bdce5e9" containerName="payload-pre-processing" containerID="cri-o://182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3" gracePeriod=30
Sep 07 10:50:02.895473 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:02.895439    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"]
Sep 07 10:50:02.900851 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:02.900798    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4e2c0-l5tbt"]
Sep 07 10:50:02.951904 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:02.951879    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:50:03.049568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.049486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxzgb\" (UniqueName: \"kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb\") pod \"655e96af-ef99-4d75-9d40-78931bdce5e9\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") "
Sep 07 10:50:03.049568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.049550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume\") pod \"655e96af-ef99-4d75-9d40-78931bdce5e9\" (UID: \"655e96af-ef99-4d75-9d40-78931bdce5e9\") "
Sep 07 10:50:03.049961 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.049926    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "655e96af-ef99-4d75-9d40-78931bdce5e9" (UID: "655e96af-ef99-4d75-9d40-78931bdce5e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:50:03.051748 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.051727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb" (OuterVolumeSpecName: "kube-api-access-rxzgb") pod "655e96af-ef99-4d75-9d40-78931bdce5e9" (UID: "655e96af-ef99-4d75-9d40-78931bdce5e9"). InnerVolumeSpecName "kube-api-access-rxzgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:50:03.150195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.150148    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzgb\" (UniqueName: \"kubernetes.io/projected/655e96af-ef99-4d75-9d40-78931bdce5e9-kube-api-access-rxzgb\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:03.150195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.150186    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/655e96af-ef99-4d75-9d40-78931bdce5e9-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:03.607216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.607136    2562 generic.go:356] "Generic (PLEG): container finished" podID="655e96af-ef99-4d75-9d40-78931bdce5e9" containerID="182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3" exitCode=0
Sep 07 10:50:03.607216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.607184    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"
Sep 07 10:50:03.607430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.607185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" event={"ID":"655e96af-ef99-4d75-9d40-78931bdce5e9","Type":"ContainerDied","Data":"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"}
Sep 07 10:50:03.607430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.607342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc" event={"ID":"655e96af-ef99-4d75-9d40-78931bdce5e9","Type":"ContainerDied","Data":"09347ba4e26279884a5b73b0be38b285e03324bef8ed28f7b393c3e21f800102"}
Sep 07 10:50:03.607430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.607379    2562 scope.go:117] "RemoveContainer" containerID="182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"
Sep 07 10:50:03.615982 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.615964    2562 scope.go:117] "RemoveContainer" containerID="182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"
Sep 07 10:50:03.616250 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:50:03.616233    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3\": container with ID starting with 182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3 not found: ID does not exist" containerID="182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"
Sep 07 10:50:03.616410 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.616260    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3"} err="failed to get container status \"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3\": rpc error: code = NotFound desc = could not find container \"182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3\": container with ID starting with 182b74f74ac4eae935a51956bee288643a5f09e905dae28e07d678f8847f97e3 not found: ID does not exist"
Sep 07 10:50:03.628274 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.628252    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:50:03.632541 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:03.632521    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4e2c0-7c7b84b46-9f5qc"]
Sep 07 10:50:04.496103 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:04.496071    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655e96af-ef99-4d75-9d40-78931bdce5e9" path="/var/lib/kubelet/pods/655e96af-ef99-4d75-9d40-78931bdce5e9/volumes"
Sep 07 10:50:04.496489 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:04.496384    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f29d70-f3f3-4ced-b56d-9535ba84092c" path="/var/lib/kubelet/pods/b4f29d70-f3f3-4ced-b56d-9535ba84092c/volumes"
Sep 07 10:50:09.698023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.697985    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"]
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698359    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655e96af-ef99-4d75-9d40-78931bdce5e9" containerName="payload-pre-processing"
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="655e96af-ef99-4d75-9d40-78931bdce5e9" containerName="payload-pre-processing"
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698389    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f29d70-f3f3-4ced-b56d-9535ba84092c" containerName="revoke-keys"
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698394    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f29d70-f3f3-4ced-b56d-9535ba84092c" containerName="revoke-keys"
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698402    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" containerName="main"
Sep 07 10:50:09.698449 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698408    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" containerName="main"
Sep 07 10:50:09.698644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698471    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f29d70-f3f3-4ced-b56d-9535ba84092c" containerName="revoke-keys"
Sep 07 10:50:09.698644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698478    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="655e96af-ef99-4d75-9d40-78931bdce5e9" containerName="payload-pre-processing"
Sep 07 10:50:09.698644 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.698484    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5fc48fc-bdc0-44e8-8c49-be3b45091de7" containerName="main"
Sep 07 10:50:09.701519 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.701504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:09.704062 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.704044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:50:09.708292 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.708256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"]
Sep 07 10:50:09.808373 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.808336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q\" (UID: \"1065c673-d268-4034-b25d-35e00a4e7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:09.909319 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.909277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q\" (UID: \"1065c673-d268-4034-b25d-35e00a4e7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:09.909873 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:09.909852    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q\" (UID: \"1065c673-d268-4034-b25d-35e00a4e7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:10.013183 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:10.013096    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:10.284015 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:10.283971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"]
Sep 07 10:50:10.286205 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:50:10.286178    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1065c673_d268_4034_b25d_35e00a4e7939.slice/crio-b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8 WatchSource:0}: Error finding container b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8: Status 404 returned error can't find the container with id b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8
Sep 07 10:50:10.646361 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:10.646332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q" event={"ID":"1065c673-d268-4034-b25d-35e00a4e7939","Type":"ContainerStarted","Data":"aa7c7b1558d6c09c2d984d69da2d785fe61e1c1b584ddf884b77311e1df58fd1"}
Sep 07 10:50:10.646518 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:10.646389    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q" event={"ID":"1065c673-d268-4034-b25d-35e00a4e7939","Type":"ContainerStarted","Data":"b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8"}
Sep 07 10:50:10.662324 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:10.662260    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q" podStartSLOduration=1.662246561 podStartE2EDuration="1.662246561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:50:10.661293519 +0000 UTC m=+1798.671946827" watchObservedRunningTime="2026-09-07 10:50:10.662246561 +0000 UTC m=+1798.672899874"
Sep 07 10:50:11.651892 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:11.651863    2562 generic.go:356] "Generic (PLEG): container finished" podID="1065c673-d268-4034-b25d-35e00a4e7939" containerID="aa7c7b1558d6c09c2d984d69da2d785fe61e1c1b584ddf884b77311e1df58fd1" exitCode=0
Sep 07 10:50:11.652295 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:11.651953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q" event={"ID":"1065c673-d268-4034-b25d-35e00a4e7939","Type":"ContainerDied","Data":"aa7c7b1558d6c09c2d984d69da2d785fe61e1c1b584ddf884b77311e1df58fd1"}
Sep 07 10:50:12.531033 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:12.531007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:50:12.533266 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:12.533243    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:50:12.783283 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:12.783220    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:12.936699 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:12.936660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca\") pod \"1065c673-d268-4034-b25d-35e00a4e7939\" (UID: \"1065c673-d268-4034-b25d-35e00a4e7939\") "
Sep 07 10:50:12.937012 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:12.936989    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1065c673-d268-4034-b25d-35e00a4e7939" (UID: "1065c673-d268-4034-b25d-35e00a4e7939"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:50:13.038146 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:13.038068    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1065c673-d268-4034-b25d-35e00a4e7939-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:13.662257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:13.662229    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"
Sep 07 10:50:13.662257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:13.662248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q" event={"ID":"1065c673-d268-4034-b25d-35e00a4e7939","Type":"ContainerDied","Data":"b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8"}
Sep 07 10:50:13.662477 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:13.662290    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f8f8f73e7ea9fad8c811cf26ed5eac4a8b200683094205361bd5198f2b23f8"
Sep 07 10:50:16.726004 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.725973    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:50:16.726387 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.726328    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1065c673-d268-4034-b25d-35e00a4e7939" containerName="revoke-keys"
Sep 07 10:50:16.726387 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.726344    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1065c673-d268-4034-b25d-35e00a4e7939" containerName="revoke-keys"
Sep 07 10:50:16.726464 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.726415    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1065c673-d268-4034-b25d-35e00a4e7939" containerName="revoke-keys"
Sep 07 10:50:16.729205 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.729184    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.731719 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.731689    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"kube-root-ca.crt\""
Sep 07 10:50:16.732849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.732825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"default-dockercfg-8q89w\""
Sep 07 10:50:16.732983 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.732962    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"rate-test-model-659410-kserve-self-signed-certs\""
Sep 07 10:50:16.733093 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.733075    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-659410\"/\"openshift-service-ca.crt\""
Sep 07 10:50:16.740676 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.740655    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:50:16.870034 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.869997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.870221 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.870062    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jqm\" (UniqueName: \"kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.870221 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.870136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.870221 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.870167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.870221 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.870191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.870434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.870268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971345 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971394    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jqm\" (UniqueName: \"kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971425    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971498 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971848 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.971964 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.971860    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.973745 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.973718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.973870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.973854    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:16.981188 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:16.981136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jqm\" (UniqueName: \"kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm\") pod \"rate-test-model-659410-kserve-5bb5744c9-sqqxc\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") " pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:17.038037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:17.038007    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:17.309129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:17.309104    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:50:17.311569 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:50:17.311542    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd39df9db_bbc7_4b2f_969f_d1c3ce9550c6.slice/crio-cb37b64227d3026c4a76426b2f09debfc42b9dc59871c123cdb989d3db489cf5 WatchSource:0}: Error finding container cb37b64227d3026c4a76426b2f09debfc42b9dc59871c123cdb989d3db489cf5: Status 404 returned error can't find the container with id cb37b64227d3026c4a76426b2f09debfc42b9dc59871c123cdb989d3db489cf5
Sep 07 10:50:17.683087 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:17.683054    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" event={"ID":"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6","Type":"ContainerStarted","Data":"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"}
Sep 07 10:50:17.683334 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:17.683105    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" event={"ID":"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6","Type":"ContainerStarted","Data":"cb37b64227d3026c4a76426b2f09debfc42b9dc59871c123cdb989d3db489cf5"}
Sep 07 10:50:17.706278 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:17.706232    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" podStartSLOduration=1.706217324 podStartE2EDuration="1.706217324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:50:17.704904635 +0000 UTC m=+1805.715557947" watchObservedRunningTime="2026-09-07 10:50:17.706217324 +0000 UTC m=+1805.716870637"
Sep 07 10:50:25.247065 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.247024    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:50:25.247520 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.247282    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" podUID="21af99b3-de35-47ba-8ca2-53238968bc08" containerName="payload-pre-processing" containerID="cri-o://3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb" gracePeriod=30
Sep 07 10:50:25.420267 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.420227    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"]
Sep 07 10:50:25.424184 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.424152    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4f3dee-sfp2q"]
Sep 07 10:50:25.493257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.493232    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:50:25.650603 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.650565    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume\") pod \"21af99b3-de35-47ba-8ca2-53238968bc08\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") "
Sep 07 10:50:25.650785 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.650645    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbfzl\" (UniqueName: \"kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl\") pod \"21af99b3-de35-47ba-8ca2-53238968bc08\" (UID: \"21af99b3-de35-47ba-8ca2-53238968bc08\") "
Sep 07 10:50:25.650960 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.650936    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "21af99b3-de35-47ba-8ca2-53238968bc08" (UID: "21af99b3-de35-47ba-8ca2-53238968bc08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:50:25.652832 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.652813    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl" (OuterVolumeSpecName: "kube-api-access-tbfzl") pod "21af99b3-de35-47ba-8ca2-53238968bc08" (UID: "21af99b3-de35-47ba-8ca2-53238968bc08"). InnerVolumeSpecName "kube-api-access-tbfzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:50:25.722184 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.722150    2562 generic.go:356] "Generic (PLEG): container finished" podID="21af99b3-de35-47ba-8ca2-53238968bc08" containerID="3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb" exitCode=0
Sep 07 10:50:25.722360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.722199    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"
Sep 07 10:50:25.722360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.722234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" event={"ID":"21af99b3-de35-47ba-8ca2-53238968bc08","Type":"ContainerDied","Data":"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"}
Sep 07 10:50:25.722360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.722288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz" event={"ID":"21af99b3-de35-47ba-8ca2-53238968bc08","Type":"ContainerDied","Data":"b639c89ac4791e68ef7e6ff8c72b0bd784ca22b8065b4c0f22970513fcc81b6a"}
Sep 07 10:50:25.722360 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.722323    2562 scope.go:117] "RemoveContainer" containerID="3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"
Sep 07 10:50:25.731293 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.731279    2562 scope.go:117] "RemoveContainer" containerID="3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"
Sep 07 10:50:25.731584 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:50:25.731568    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb\": container with ID starting with 3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb not found: ID does not exist" containerID="3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"
Sep 07 10:50:25.731662 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.731593    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb"} err="failed to get container status \"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb\": rpc error: code = NotFound desc = could not find container \"3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb\": container with ID starting with 3d8eae23cfa7da101ea574f91c143a4f7479f858f9f775d5fd7bdde010a55feb not found: ID does not exist"
Sep 07 10:50:25.745547 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.745517    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:50:25.749655 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.749635    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4f3dee-5d5c4766bb-qk5jz"]
Sep 07 10:50:25.751204 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.751187    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbfzl\" (UniqueName: \"kubernetes.io/projected/21af99b3-de35-47ba-8ca2-53238968bc08-kube-api-access-tbfzl\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:25.751255 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:25.751208    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af99b3-de35-47ba-8ca2-53238968bc08-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:26.495913 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:26.495881    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1065c673-d268-4034-b25d-35e00a4e7939" path="/var/lib/kubelet/pods/1065c673-d268-4034-b25d-35e00a4e7939/volumes"
Sep 07 10:50:26.496283 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:26.496219    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21af99b3-de35-47ba-8ca2-53238968bc08" path="/var/lib/kubelet/pods/21af99b3-de35-47ba-8ca2-53238968bc08/volumes"
Sep 07 10:50:27.038573 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:27.038539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:27.038782 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:27.038587    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:27.050909 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:27.050876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:27.742921 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:27.742894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:50:42.736456 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:42.736424    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"]
Sep 07 10:50:42.739432 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:42.739411    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a158df-29812965-jmm2r"]
Sep 07 10:50:42.777135 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:42.777102    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:50:42.777425 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:42.777372    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" podUID="958c82a1-332e-416d-9f03-7112419ca8f2" containerName="payload-pre-processing" containerID="cri-o://87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb" gracePeriod=30
Sep 07 10:50:43.034758 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.034733    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:50:43.099229 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.099194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvhbk\" (UniqueName: \"kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk\") pod \"958c82a1-332e-416d-9f03-7112419ca8f2\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") "
Sep 07 10:50:43.099405 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.099248    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume\") pod \"958c82a1-332e-416d-9f03-7112419ca8f2\" (UID: \"958c82a1-332e-416d-9f03-7112419ca8f2\") "
Sep 07 10:50:43.099620 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.099595    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "958c82a1-332e-416d-9f03-7112419ca8f2" (UID: "958c82a1-332e-416d-9f03-7112419ca8f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:50:43.101611 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.101581    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk" (OuterVolumeSpecName: "kube-api-access-gvhbk") pod "958c82a1-332e-416d-9f03-7112419ca8f2" (UID: "958c82a1-332e-416d-9f03-7112419ca8f2"). InnerVolumeSpecName "kube-api-access-gvhbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:50:43.199943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.199902    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhbk\" (UniqueName: \"kubernetes.io/projected/958c82a1-332e-416d-9f03-7112419ca8f2-kube-api-access-gvhbk\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:43.199943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.199936    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/958c82a1-332e-416d-9f03-7112419ca8f2-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:43.808348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.808291    2562 generic.go:356] "Generic (PLEG): container finished" podID="958c82a1-332e-416d-9f03-7112419ca8f2" containerID="87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb" exitCode=0
Sep 07 10:50:43.808777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.808364    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"
Sep 07 10:50:43.808777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.808391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" event={"ID":"958c82a1-332e-416d-9f03-7112419ca8f2","Type":"ContainerDied","Data":"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"}
Sep 07 10:50:43.808777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.808450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc" event={"ID":"958c82a1-332e-416d-9f03-7112419ca8f2","Type":"ContainerDied","Data":"94d175a7a37c5c3ef8c10fc973b829f6b7a34e34bd1119637cfecac9941cecae"}
Sep 07 10:50:43.808777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.808474    2562 scope.go:117] "RemoveContainer" containerID="87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"
Sep 07 10:50:43.817258 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.817242    2562 scope.go:117] "RemoveContainer" containerID="87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"
Sep 07 10:50:43.817588 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:50:43.817569    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb\": container with ID starting with 87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb not found: ID does not exist" containerID="87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"
Sep 07 10:50:43.817677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.817601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb"} err="failed to get container status \"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb\": rpc error: code = NotFound desc = could not find container \"87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb\": container with ID starting with 87119a6d13977829791c4647f318ee2fc06975e360438ab5d82ab09a67c383bb not found: ID does not exist"
Sep 07 10:50:43.828777 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.828750    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:50:43.832684 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:43.832663    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-a158df-54d474f8f8-hjnqc"]
Sep 07 10:50:44.496241 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:44.496206    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69044d71-a3fd-4331-ac7b-1bf1f15935f4" path="/var/lib/kubelet/pods/69044d71-a3fd-4331-ac7b-1bf1f15935f4/volumes"
Sep 07 10:50:44.496636 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:44.496616    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958c82a1-332e-416d-9f03-7112419ca8f2" path="/var/lib/kubelet/pods/958c82a1-332e-416d-9f03-7112419ca8f2/volumes"
Sep 07 10:50:49.334878 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:49.334837    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:50:49.335406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:49.335043    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy" containerID="cri-o://7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1" gracePeriod=30
Sep 07 10:50:54.584129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.584103    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:50:54.698789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.698789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.698789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698765    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.698789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698789    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698807    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzxp7\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698832    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698847    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698924    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.698985    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy\") pod \"3fabef64-950d-4c7b-9c10-da0321281d9f\" (UID: \"3fabef64-950d-4c7b-9c10-da0321281d9f\") "
Sep 07 10:50:54.699118 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:50:54.699469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:50:54.699469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699260    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:50:54.699469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699268    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:50:54.699469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699335    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.699469 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699356    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-workload-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.699743 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.699548    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:50:54.701365 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.701343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:50:54.701490 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.701411    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7" (OuterVolumeSpecName: "kube-api-access-hzxp7") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "kube-api-access-hzxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:50:54.701748 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.701728    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 10:50:54.701816 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.701743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3fabef64-950d-4c7b-9c10-da0321281d9f" (UID: "3fabef64-950d-4c7b-9c10-da0321281d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:50:54.800749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800709    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-data\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.800749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800741    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-envoy\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.800749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800749    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fabef64-950d-4c7b-9c10-da0321281d9f-istiod-ca-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.800749 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800758    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fabef64-950d-4c7b-9c10-da0321281d9f-credential-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.801023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800767    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxp7\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-kube-api-access-hzxp7\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.801023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800776    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.801023 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.800784    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fabef64-950d-4c7b-9c10-da0321281d9f-istio-podinfo\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:50:54.862849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.862807    2562 generic.go:356] "Generic (PLEG): container finished" podID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerID="7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1" exitCode=0
Sep 07 10:50:54.863004 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.862859    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"
Sep 07 10:50:54.863004 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.862888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerDied","Data":"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"}
Sep 07 10:50:54.863004 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.862945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9" event={"ID":"3fabef64-950d-4c7b-9c10-da0321281d9f","Type":"ContainerDied","Data":"45e26468dc0782f23ff8b6b77ffde259085412ba738bb8a848100477f095494c"}
Sep 07 10:50:54.863591 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.862977    2562 scope.go:117] "RemoveContainer" containerID="7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"
Sep 07 10:50:54.872845 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.872827    2562 scope.go:117] "RemoveContainer" containerID="70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"
Sep 07 10:50:54.883111 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.883087    2562 scope.go:117] "RemoveContainer" containerID="7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"
Sep 07 10:50:54.883748 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:50:54.883719    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1\": container with ID starting with 7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1 not found: ID does not exist" containerID="7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"
Sep 07 10:50:54.883845 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.883760    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1"} err="failed to get container status \"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1\": rpc error: code = NotFound desc = could not find container \"7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1\": container with ID starting with 7448c54ed2bf85667bacfd4f4fbbf7b5c0049ad9d21a1f39d23690a0b7faddc1 not found: ID does not exist"
Sep 07 10:50:54.883845 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.883787    2562 scope.go:117] "RemoveContainer" containerID="70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"
Sep 07 10:50:54.884196 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:50:54.884175    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6\": container with ID starting with 70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6 not found: ID does not exist" containerID="70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"
Sep 07 10:50:54.884260 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.884205    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6"} err="failed to get container status \"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6\": rpc error: code = NotFound desc = could not find container \"70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6\": container with ID starting with 70ac554162730e9dac852770cec46e2ade469bef8e1aff42e645f374e79d72a6 not found: ID does not exist"
Sep 07 10:50:54.884947 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.884928    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:50:54.888717 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:54.888695    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a158df-openshift-default-7c6cbdf976-trkv9"]
Sep 07 10:50:56.495396 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:50:56.495358    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" path="/var/lib/kubelet/pods/3fabef64-950d-4c7b-9c10-da0321281d9f/volumes"
Sep 07 10:51:00.375170 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.375135    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:51:00.375690 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.375371    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" podUID="65419d0a-84a3-4288-a2b6-a6f49068669d" containerName="payload-processing" containerID="cri-o://7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe" gracePeriod=30
Sep 07 10:51:00.631685 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.631623    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:51:00.756351 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.756287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume\") pod \"65419d0a-84a3-4288-a2b6-a6f49068669d\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") "
Sep 07 10:51:00.756534 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.756402    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmpsl\" (UniqueName: \"kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl\") pod \"65419d0a-84a3-4288-a2b6-a6f49068669d\" (UID: \"65419d0a-84a3-4288-a2b6-a6f49068669d\") "
Sep 07 10:51:00.756747 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.756717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65419d0a-84a3-4288-a2b6-a6f49068669d" (UID: "65419d0a-84a3-4288-a2b6-a6f49068669d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:51:00.758689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.758663    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl" (OuterVolumeSpecName: "kube-api-access-vmpsl") pod "65419d0a-84a3-4288-a2b6-a6f49068669d" (UID: "65419d0a-84a3-4288-a2b6-a6f49068669d"). InnerVolumeSpecName "kube-api-access-vmpsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:51:00.857211 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.857171    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmpsl\" (UniqueName: \"kubernetes.io/projected/65419d0a-84a3-4288-a2b6-a6f49068669d-kube-api-access-vmpsl\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:00.857211 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.857201    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65419d0a-84a3-4288-a2b6-a6f49068669d-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:00.891664 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.891573    2562 generic.go:356] "Generic (PLEG): container finished" podID="65419d0a-84a3-4288-a2b6-a6f49068669d" containerID="7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe" exitCode=0
Sep 07 10:51:00.891664 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.891621    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"
Sep 07 10:51:00.891841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.891663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" event={"ID":"65419d0a-84a3-4288-a2b6-a6f49068669d","Type":"ContainerDied","Data":"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"}
Sep 07 10:51:00.891841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.891713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm" event={"ID":"65419d0a-84a3-4288-a2b6-a6f49068669d","Type":"ContainerDied","Data":"bd10ed59a05b3409b4cf65d0ad104349eb50c4f865c83f8e15084d2c1a6120c1"}
Sep 07 10:51:00.891841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.891735    2562 scope.go:117] "RemoveContainer" containerID="7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"
Sep 07 10:51:00.907255 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.907232    2562 scope.go:117] "RemoveContainer" containerID="7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"
Sep 07 10:51:00.907595 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:51:00.907574    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe\": container with ID starting with 7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe not found: ID does not exist" containerID="7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"
Sep 07 10:51:00.907652 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.907605    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe"} err="failed to get container status \"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe\": rpc error: code = NotFound desc = could not find container \"7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe\": container with ID starting with 7dd957fc8b0617b547c57e49f930c554c902a0d61336d3d3a17c4ed4a3e204fe not found: ID does not exist"
Sep 07 10:51:00.915998 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.915969    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:51:00.922383 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:00.922357    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5e934f-bf6cb7c47-lswwm"]
Sep 07 10:51:02.495656 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:02.495622    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65419d0a-84a3-4288-a2b6-a6f49068669d" path="/var/lib/kubelet/pods/65419d0a-84a3-4288-a2b6-a6f49068669d/volumes"
Sep 07 10:51:12.662021 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:12.661987    2562 scope.go:117] "RemoveContainer" containerID="42ecc7e97cec3cc7de72abd44cd2768b71c88d4fbd9c8b5ccfbb4f70ad4ec5f2"
Sep 07 10:51:24.350948 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:24.350915    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:51:25.308214 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:25.308176    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:51:30.653083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653042    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653396    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653410    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653417    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653422    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653437    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21af99b3-de35-47ba-8ca2-53238968bc08" containerName="payload-pre-processing"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653444    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="21af99b3-de35-47ba-8ca2-53238968bc08" containerName="payload-pre-processing"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653458    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958c82a1-332e-416d-9f03-7112419ca8f2" containerName="payload-pre-processing"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653463    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="958c82a1-332e-416d-9f03-7112419ca8f2" containerName="payload-pre-processing"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653468    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65419d0a-84a3-4288-a2b6-a6f49068669d" containerName="payload-processing"
Sep 07 10:51:30.653505 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653472    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="65419d0a-84a3-4288-a2b6-a6f49068669d" containerName="payload-processing"
Sep 07 10:51:30.653821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653520    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="21af99b3-de35-47ba-8ca2-53238968bc08" containerName="payload-pre-processing"
Sep 07 10:51:30.653821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653527    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653534    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fabef64-950d-4c7b-9c10-da0321281d9f" containerName="istio-proxy"
Sep 07 10:51:30.653821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653539    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="958c82a1-332e-416d-9f03-7112419ca8f2" containerName="payload-pre-processing"
Sep 07 10:51:30.653821 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.653544    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="65419d0a-84a3-4288-a2b6-a6f49068669d" containerName="payload-processing"
Sep 07 10:51:30.656501 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.656485    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.659122 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.659098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bef02f-dockercfg-wrb2l\""
Sep 07 10:51:30.659253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.659157    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bef02f\""
Sep 07 10:51:30.667098 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.667075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:51:30.713740 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.713696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f65df\" (UniqueName: \"kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.713920 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.713753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.814382 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.814346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f65df\" (UniqueName: \"kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.814565 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.814401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.815045 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.815027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.824283 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.824248    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f65df\" (UniqueName: \"kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df\") pod \"payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:30.967629 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:30.967529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:31.239363 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:31.239275    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:51:31.241752 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:51:31.241722    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9b65b3_21e2_47c0_9dac_b876598b8d3a.slice/crio-3e5b9d9637bbad8e6cfb772deefcf08623cdafbf8e02c9b1717ffe6e526156e6 WatchSource:0}: Error finding container 3e5b9d9637bbad8e6cfb772deefcf08623cdafbf8e02c9b1717ffe6e526156e6: Status 404 returned error can't find the container with id 3e5b9d9637bbad8e6cfb772deefcf08623cdafbf8e02c9b1717ffe6e526156e6
Sep 07 10:51:32.034000 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:32.033964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" event={"ID":"9b9b65b3-21e2-47c0-9dac-b876598b8d3a","Type":"ContainerStarted","Data":"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"}
Sep 07 10:51:32.034514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:32.034013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" event={"ID":"9b9b65b3-21e2-47c0-9dac-b876598b8d3a","Type":"ContainerStarted","Data":"3e5b9d9637bbad8e6cfb772deefcf08623cdafbf8e02c9b1717ffe6e526156e6"}
Sep 07 10:51:32.034514 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:32.034101    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:32.052425 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:32.052372    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" podStartSLOduration=2.052354763 podStartE2EDuration="2.052354763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:51:32.051663836 +0000 UTC m=+1880.062317152" watchObservedRunningTime="2026-09-07 10:51:32.052354763 +0000 UTC m=+1880.063008135"
Sep 07 10:51:39.371459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.371362    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" podUID="fe609053-da73-4ab6-8a61-7649a4f5bb08" containerName="main" containerID="cri-o://2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42" gracePeriod=45
Sep 07 10:51:39.615971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.615944    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:51:39.694475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694370    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.694475 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694420    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.694711 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.694711 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694506    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.694711 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694548    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.694711 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.694598    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr877\" (UniqueName: \"kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877\") pod \"fe609053-da73-4ab6-8a61-7649a4f5bb08\" (UID: \"fe609053-da73-4ab6-8a61-7649a4f5bb08\") "
Sep 07 10:51:39.695677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.695088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:39.695677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.695235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home" (OuterVolumeSpecName: "home") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:39.695677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.695248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:39.698059 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.697702    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:51:39.700352 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.698290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877" (OuterVolumeSpecName: "kube-api-access-lr877") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "kube-api-access-lr877". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:51:39.703389 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.703348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm" (OuterVolumeSpecName: "dshm") pod "fe609053-da73-4ab6-8a61-7649a4f5bb08" (UID: "fe609053-da73-4ab6-8a61-7649a4f5bb08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:39.796129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796075    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:39.796129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796122    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr877\" (UniqueName: \"kubernetes.io/projected/fe609053-da73-4ab6-8a61-7649a4f5bb08-kube-api-access-lr877\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:39.796129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796133    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe609053-da73-4ab6-8a61-7649a4f5bb08-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:39.796129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796141    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:39.796129 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796150    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:39.796479 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:39.796158    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe609053-da73-4ab6-8a61-7649a4f5bb08-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.069688 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.069654    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe609053-da73-4ab6-8a61-7649a4f5bb08" containerID="2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42" exitCode=0
Sep 07 10:51:40.069877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.069714    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" event={"ID":"fe609053-da73-4ab6-8a61-7649a4f5bb08","Type":"ContainerDied","Data":"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"}
Sep 07 10:51:40.069877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.069716    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"
Sep 07 10:51:40.069877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.069746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7" event={"ID":"fe609053-da73-4ab6-8a61-7649a4f5bb08","Type":"ContainerDied","Data":"ea30da70d0e972952f911a690292e46f078978ba045d6b8975a7d7a35180ed06"}
Sep 07 10:51:40.069877 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.069763    2562 scope.go:117] "RemoveContainer" containerID="2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"
Sep 07 10:51:40.078713 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.078693    2562 scope.go:117] "RemoveContainer" containerID="2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"
Sep 07 10:51:40.079030 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:51:40.079005    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42\": container with ID starting with 2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42 not found: ID does not exist" containerID="2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"
Sep 07 10:51:40.079097 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.079039    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42"} err="failed to get container status \"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42\": rpc error: code = NotFound desc = could not find container \"2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42\": container with ID starting with 2c86a26c85afbce0335ac50c9d1a87904d7134210eeea99a7147d274e5417d42 not found: ID does not exist"
Sep 07 10:51:40.090901 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.090863    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:51:40.096564 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.096535    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5a0ff8/rate-test-model-5a0ff8-kserve-6557df9f-qxqv7"]
Sep 07 10:51:40.330125 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.330025    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" podUID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" containerName="main" containerID="cri-o://d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785" gracePeriod=45
Sep 07 10:51:40.496233 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.496193    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe609053-da73-4ab6-8a61-7649a4f5bb08" path="/var/lib/kubelet/pods/fe609053-da73-4ab6-8a61-7649a4f5bb08/volumes"
Sep 07 10:51:40.586664 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.586595    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:51:40.705071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705032    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4jqm\" (UniqueName: \"kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705071 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705077    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705093    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705118    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705172    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705386 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home\") pod \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\" (UID: \"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6\") "
Sep 07 10:51:40.705604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705398    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:40.705604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home" (OuterVolumeSpecName: "home") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:40.705604 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705545    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:40.705739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705629    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.705739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705649    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.705739 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.705663    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.707467 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.707446    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:51:40.707563 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.707543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm" (OuterVolumeSpecName: "kube-api-access-x4jqm") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "kube-api-access-x4jqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:51:40.707609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.707544    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm" (OuterVolumeSpecName: "dshm") pod "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" (UID: "d39df9db-bbc7-4b2f-969f-d1c3ce9550c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:51:40.806108 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.806066    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.806108 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.806102    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jqm\" (UniqueName: \"kubernetes.io/projected/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-kube-api-access-x4jqm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:40.806108 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:40.806112    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:51:41.075373 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.075336    2562 generic.go:356] "Generic (PLEG): container finished" podID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" containerID="d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785" exitCode=0
Sep 07 10:51:41.075568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.075439    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"
Sep 07 10:51:41.075568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.075454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" event={"ID":"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6","Type":"ContainerDied","Data":"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"}
Sep 07 10:51:41.075568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.075503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc" event={"ID":"d39df9db-bbc7-4b2f-969f-d1c3ce9550c6","Type":"ContainerDied","Data":"cb37b64227d3026c4a76426b2f09debfc42b9dc59871c123cdb989d3db489cf5"}
Sep 07 10:51:41.075568 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.075527    2562 scope.go:117] "RemoveContainer" containerID="d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"
Sep 07 10:51:41.084406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.084384    2562 scope.go:117] "RemoveContainer" containerID="d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"
Sep 07 10:51:41.084739 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:51:41.084709    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785\": container with ID starting with d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785 not found: ID does not exist" containerID="d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"
Sep 07 10:51:41.084793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.084749    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785"} err="failed to get container status \"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785\": rpc error: code = NotFound desc = could not find container \"d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785\": container with ID starting with d499c9b4ca6f994408ab440331044c29cb95ded157f1ba5e750745bf0b9d8785 not found: ID does not exist"
Sep 07 10:51:41.096276 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.096241    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:51:41.101366 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:41.101290    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-659410/rate-test-model-659410-kserve-5bb5744c9-sqqxc"]
Sep 07 10:51:42.496215 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:42.496181    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" path="/var/lib/kubelet/pods/d39df9db-bbc7-4b2f-969f-d1c3ce9550c6/volumes"
Sep 07 10:51:43.040884 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:43.040850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:51:54.630510 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.630473    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:51:54.631009 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.630920    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe609053-da73-4ab6-8a61-7649a4f5bb08" containerName="main"
Sep 07 10:51:54.631009 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.630934    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe609053-da73-4ab6-8a61-7649a4f5bb08" containerName="main"
Sep 07 10:51:54.631009 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.630953    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" containerName="main"
Sep 07 10:51:54.631009 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.630963    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" containerName="main"
Sep 07 10:51:54.631169 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.631045    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39df9db-bbc7-4b2f-969f-d1c3ce9550c6" containerName="main"
Sep 07 10:51:54.631169 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.631058    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe609053-da73-4ab6-8a61-7649a4f5bb08" containerName="main"
Sep 07 10:51:54.633726 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.633706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.636362 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.636334    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-738923-openshift-default-dockercfg-rpg8f\""
Sep 07 10:51:54.636490 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.636433    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 10:51:54.645728 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.645697    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:51:54.736174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736174 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whn2l\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736494 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.736712 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.736509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.837971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.837929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.837971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.837975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838030    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whn2l\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838256 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838681 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838681 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.838681 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838639    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.839008 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.838907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.839151 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.839131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.840766 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.840746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.840868 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.840775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.846993 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.846930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whn2l\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.847229 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.847209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token\") pod \"e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") " pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:54.944230 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:54.944128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:55.221294 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:55.221207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:51:55.225249 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:51:55.225222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb243a441_6c46_4126_b3c8_93c698466b12.slice/crio-a95ad8992494b06211898696c2042c6fe442b82cd327cca0a0ba3bbe67d38972 WatchSource:0}: Error finding container a95ad8992494b06211898696c2042c6fe442b82cd327cca0a0ba3bbe67d38972: Status 404 returned error can't find the container with id a95ad8992494b06211898696c2042c6fe442b82cd327cca0a0ba3bbe67d38972
Sep 07 10:51:55.227936 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:55.227901    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:51:55.227997 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:55.227986    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:51:55.228043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:55.228029    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:51:56.145101 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:56.145061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerStarted","Data":"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"}
Sep 07 10:51:56.145535 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:56.145122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerStarted","Data":"a95ad8992494b06211898696c2042c6fe442b82cd327cca0a0ba3bbe67d38972"}
Sep 07 10:51:56.169393 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:56.169316    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podStartSLOduration=2.1692789550000002 podStartE2EDuration="2.169278955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:51:56.166471628 +0000 UTC m=+1904.177124956" watchObservedRunningTime="2026-09-07 10:51:56.169278955 +0000 UTC m=+1904.179932272"
Sep 07 10:51:56.945034 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:56.944993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:56.950172 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:56.950144    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:57.150791 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:57.150760    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:57.151870 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:57.151848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:51:59.930738 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:59.930700    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:51:59.934138 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:59.934120    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:51:59.936841 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:59.936817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-738923-dockercfg-9knbx\""
Sep 07 10:51:59.937085 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:59.937069    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-738923\""
Sep 07 10:51:59.944992 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:51:59.944967    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:52:00.090761 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.090723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.090943 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.090859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5z5\" (UniqueName: \"kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.192049 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.191940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.192049 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.192043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5z5\" (UniqueName: \"kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.192779 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.192758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.201589 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.201563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5z5\" (UniqueName: \"kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5\") pod \"payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.245442 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.245402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:00.516788 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:00.516761    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:52:00.518984 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:52:00.518959    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a2324b_cadd_4c76_a5a0_70a35fa40ec9.slice/crio-1b80d71babaf2ea996303a128c6d6b1d3627aa87796e58ff2ebd311e7482ed6e WatchSource:0}: Error finding container 1b80d71babaf2ea996303a128c6d6b1d3627aa87796e58ff2ebd311e7482ed6e: Status 404 returned error can't find the container with id 1b80d71babaf2ea996303a128c6d6b1d3627aa87796e58ff2ebd311e7482ed6e
Sep 07 10:52:01.171348 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:01.171313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" event={"ID":"89a2324b-cadd-4c76-a5a0-70a35fa40ec9","Type":"ContainerStarted","Data":"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"}
Sep 07 10:52:01.171793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:01.171413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" event={"ID":"89a2324b-cadd-4c76-a5a0-70a35fa40ec9","Type":"ContainerStarted","Data":"1b80d71babaf2ea996303a128c6d6b1d3627aa87796e58ff2ebd311e7482ed6e"}
Sep 07 10:52:01.171793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:01.171498    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:01.187985 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:01.187926    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" podStartSLOduration=2.18791139 podStartE2EDuration="2.18791139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:52:01.186272703 +0000 UTC m=+1909.196926021" watchObservedRunningTime="2026-09-07 10:52:01.18791139 +0000 UTC m=+1909.198564741"
Sep 07 10:52:12.179247 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:12.179217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:52:12.965646 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:12.965602    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:35112->10.133.0.62:15021: read: connection reset by peer" start-of-body=
Sep 07 10:52:12.965959 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:12.965676    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": read tcp 10.133.0.2:35112->10.133.0.62:15021: read: connection reset by peer"
Sep 07 10:52:13.234897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.234796    2562 generic.go:356] "Generic (PLEG): container finished" podID="b243a441-6c46-4126-b3c8-93c698466b12" containerID="35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2" exitCode=0
Sep 07 10:52:13.234897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.234869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerDied","Data":"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"}
Sep 07 10:52:13.235285 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.235264    2562 scope.go:117] "RemoveContainer" containerID="35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"
Sep 07 10:52:13.236322 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.236280    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:52:13.236383 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.236359    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:52:13.236419 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:13.236389    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 07 10:52:14.240668 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:14.240634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerStarted","Data":"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"}
Sep 07 10:52:14.944950 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:14.944896    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:52:14.945279 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:14.945249    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 10:52:14.945432 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:14.945353    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 10:52:15.945035 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:15.944994    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 10:52:15.945552 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:15.945072    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 10:52:16.944793 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:16.944747    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 10:52:16.944980 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:16.944810    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 10:52:17.944780 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:17.944743    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 10:52:17.945179 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:17.944804    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 10:52:18.945434 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:18.945393    2562 patch_prober.go:28] interesting pod/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 07 10:52:18.945849 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:18.945454    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 07 10:52:19.948732 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:19.948701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:52:19.949154 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:19.948962    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:52:19.949744 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:19.949725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:52:28.244180 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:28.244152    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm_89a2324b-cadd-4c76-a5a0-70a35fa40ec9/payload-processing/0.log"
Sep 07 10:52:37.124195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.124156    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:52:37.128888 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.128868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.131805 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.131784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bef02f\"/\"kube-root-ca.crt\""
Sep 07 10:52:37.133111 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.133088    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bef02f\"/\"openshift-service-ca.crt\""
Sep 07 10:52:37.133111 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.133107    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bef02f\"/\"ipp-route-bef02f-kserve-self-signed-certs\""
Sep 07 10:52:37.133273 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.133090    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bef02f\"/\"default-dockercfg-lnw2l\""
Sep 07 10:52:37.136575 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.136553    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:52:37.215691 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.215842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.215842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.215842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.215842 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znsw\" (UniqueName: \"kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.216031 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.215878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317075 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317219 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317219 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317219 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317418 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znsw\" (UniqueName: \"kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317418 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317524 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317585 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.317696 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.317676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.319580 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.319557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.319904 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.319885    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.326814 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.326796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znsw\" (UniqueName: \"kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw\") pod \"ipp-route-bef02f-kserve-6555cb6bd7-4zx2t\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") " pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.440598 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.440517    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:37.711462 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:37.711439    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:52:37.713732 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:52:37.713707    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf945369_7c41_4fd5_8d46_e83864f8efa6.slice/crio-79b5ddb665a64ce60a3d5db270d64ba45efae6b7fa20e4dc61962e96dfc9cc72 WatchSource:0}: Error finding container 79b5ddb665a64ce60a3d5db270d64ba45efae6b7fa20e4dc61962e96dfc9cc72: Status 404 returned error can't find the container with id 79b5ddb665a64ce60a3d5db270d64ba45efae6b7fa20e4dc61962e96dfc9cc72
Sep 07 10:52:38.347616 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:38.347581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" event={"ID":"af945369-7c41-4fd5-8d46-e83864f8efa6","Type":"ContainerStarted","Data":"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"}
Sep 07 10:52:38.347972 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:38.347624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" event={"ID":"af945369-7c41-4fd5-8d46-e83864f8efa6","Type":"ContainerStarted","Data":"79b5ddb665a64ce60a3d5db270d64ba45efae6b7fa20e4dc61962e96dfc9cc72"}
Sep 07 10:52:38.366044 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:38.365998    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" podStartSLOduration=1.3659842420000001 podStartE2EDuration="1.365984242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:52:38.364329836 +0000 UTC m=+1946.374983151" watchObservedRunningTime="2026-09-07 10:52:38.365984242 +0000 UTC m=+1946.376637557"
Sep 07 10:52:47.441587 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:47.441552    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:47.441974 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:47.441629    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:47.453889 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:47.453864    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:52:48.410052 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:52:48.410018    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:53:06.148602 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:06.148573    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm_89a2324b-cadd-4c76-a5a0-70a35fa40ec9/payload-processing/0.log"
Sep 07 10:53:11.143955 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.143910    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:11.147725 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.147700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.150398 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.150377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-39c953-dockercfg-trrt4\""
Sep 07 10:53:11.150509 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.150377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-39c953\""
Sep 07 10:53:11.157947 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.157916    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:11.200837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.200801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.200837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.200836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthgc\" (UniqueName: \"kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.301698 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.301655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.301698 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.301698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthgc\" (UniqueName: \"kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.302430 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.302410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.310372 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.310351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthgc\" (UniqueName: \"kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc\") pod \"payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.459837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.459751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:11.731379 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:11.731353    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:11.733278 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:53:11.733245    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ffcb36_c388_4404_9f3e_db0bcb475b8e.slice/crio-a6fef3749359b614b361f04257fac5690e2a07ba77b1cd069815e87e8e120d87 WatchSource:0}: Error finding container a6fef3749359b614b361f04257fac5690e2a07ba77b1cd069815e87e8e120d87: Status 404 returned error can't find the container with id a6fef3749359b614b361f04257fac5690e2a07ba77b1cd069815e87e8e120d87
Sep 07 10:53:12.510167 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:12.510130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" event={"ID":"e6ffcb36-c388-4404-9f3e-db0bcb475b8e","Type":"ContainerStarted","Data":"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"}
Sep 07 10:53:12.510571 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:12.510190    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" event={"ID":"e6ffcb36-c388-4404-9f3e-db0bcb475b8e","Type":"ContainerStarted","Data":"a6fef3749359b614b361f04257fac5690e2a07ba77b1cd069815e87e8e120d87"}
Sep 07 10:53:12.510571 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:12.510233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:12.527183 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:12.527137    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" podStartSLOduration=1.527122836 podStartE2EDuration="1.527122836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:53:12.526544902 +0000 UTC m=+1980.537198217" watchObservedRunningTime="2026-09-07 10:53:12.527122836 +0000 UTC m=+1980.537776150"
Sep 07 10:53:23.516903 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:23.516869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:35.108364 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.108328    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"]
Sep 07 10:53:35.111574 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.111557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:35.113932 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.113907    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:53:35.119253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.119229    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"]
Sep 07 10:53:35.211341 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.211294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v\" (UID: \"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:35.312609 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.312576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v\" (UID: \"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:35.313226 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.313204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v\" (UID: \"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:35.423716 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.423639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:35.694742 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:35.694717    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"]
Sep 07 10:53:35.696631 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:53:35.696601    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5fd56c_3d97_4c2a_ac90_1a435c6d8827.slice/crio-c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2 WatchSource:0}: Error finding container c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2: Status 404 returned error can't find the container with id c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2
Sep 07 10:53:36.623692 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:36.623661    2562 generic.go:356] "Generic (PLEG): container finished" podID="6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" containerID="a1239b5dba0ef7576e34c9c1faf5291439d87ee47cb29b5647e8edffc2157fe6" exitCode=0
Sep 07 10:53:36.624043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:36.623740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v" event={"ID":"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827","Type":"ContainerDied","Data":"a1239b5dba0ef7576e34c9c1faf5291439d87ee47cb29b5647e8edffc2157fe6"}
Sep 07 10:53:36.624043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:36.623787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v" event={"ID":"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827","Type":"ContainerStarted","Data":"c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2"}
Sep 07 10:53:37.753946 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:37.753925    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:37.833971 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:37.833941    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca\") pod \"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827\" (UID: \"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827\") "
Sep 07 10:53:37.834272 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:37.834250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" (UID: "6a5fd56c-3d97-4c2a-ac90-1a435c6d8827"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:53:37.934969 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:37.934903    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:53:38.633103 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:38.633079    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"
Sep 07 10:53:38.633289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:38.633075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v" event={"ID":"6a5fd56c-3d97-4c2a-ac90-1a435c6d8827","Type":"ContainerDied","Data":"c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2"}
Sep 07 10:53:38.633289 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:38.633204    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2dfc6f11eb0834ccefa7f99572691eb1d5f3fe366e8f29f6388d9902bd1e4b2"
Sep 07 10:53:41.081084 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.081052    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:41.081559 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.081334    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" podUID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" containerName="payload-pre-processing" containerID="cri-o://6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c" gracePeriod=30
Sep 07 10:53:41.243493 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.243396    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"]
Sep 07 10:53:41.246789 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.246763    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39c953-77b9v"]
Sep 07 10:53:41.326743 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.326715    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:41.462351 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.462244    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jthgc\" (UniqueName: \"kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc\") pod \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") "
Sep 07 10:53:41.462504 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.462388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume\") pod \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\" (UID: \"e6ffcb36-c388-4404-9f3e-db0bcb475b8e\") "
Sep 07 10:53:41.462738 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.462708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6ffcb36-c388-4404-9f3e-db0bcb475b8e" (UID: "e6ffcb36-c388-4404-9f3e-db0bcb475b8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:53:41.464569 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.464535    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc" (OuterVolumeSpecName: "kube-api-access-jthgc") pod "e6ffcb36-c388-4404-9f3e-db0bcb475b8e" (UID: "e6ffcb36-c388-4404-9f3e-db0bcb475b8e"). InnerVolumeSpecName "kube-api-access-jthgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:53:41.563574 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.563540    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthgc\" (UniqueName: \"kubernetes.io/projected/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-kube-api-access-jthgc\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:53:41.563574 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.563573    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ffcb36-c388-4404-9f3e-db0bcb475b8e-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:53:41.647491 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.647458    2562 generic.go:356] "Generic (PLEG): container finished" podID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" containerID="6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c" exitCode=0
Sep 07 10:53:41.647628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.647502    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"
Sep 07 10:53:41.647628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.647542    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" event={"ID":"e6ffcb36-c388-4404-9f3e-db0bcb475b8e","Type":"ContainerDied","Data":"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"}
Sep 07 10:53:41.647628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.647598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w" event={"ID":"e6ffcb36-c388-4404-9f3e-db0bcb475b8e","Type":"ContainerDied","Data":"a6fef3749359b614b361f04257fac5690e2a07ba77b1cd069815e87e8e120d87"}
Sep 07 10:53:41.647628 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.647620    2562 scope.go:117] "RemoveContainer" containerID="6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"
Sep 07 10:53:41.656092 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.656077    2562 scope.go:117] "RemoveContainer" containerID="6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"
Sep 07 10:53:41.656416 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:53:41.656394    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c\": container with ID starting with 6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c not found: ID does not exist" containerID="6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"
Sep 07 10:53:41.656504 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.656428    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c"} err="failed to get container status \"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c\": rpc error: code = NotFound desc = could not find container \"6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c\": container with ID starting with 6cfe27d12a5c2629d6cfe77e4e4c4802fbb97ef68e1502255069b833d498dc8c not found: ID does not exist"
Sep 07 10:53:41.668420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.668397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:41.674677 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:41.674658    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-39c953-7cdcdd4966-kjl9w"]
Sep 07 10:53:42.496054 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:42.496022    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" path="/var/lib/kubelet/pods/6a5fd56c-3d97-4c2a-ac90-1a435c6d8827/volumes"
Sep 07 10:53:42.496490 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:53:42.496398    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" path="/var/lib/kubelet/pods/e6ffcb36-c388-4404-9f3e-db0bcb475b8e/volumes"
Sep 07 10:54:07.846575 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:07.846540    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:54:07.847203 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:07.846781    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" podUID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" containerName="payload-pre-processing" containerID="cri-o://8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7" gracePeriod=30
Sep 07 10:54:08.088176 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.088154    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:54:08.183043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.182961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume\") pod \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") "
Sep 07 10:54:08.183193 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.183060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f65df\" (UniqueName: \"kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df\") pod \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\" (UID: \"9b9b65b3-21e2-47c0-9dac-b876598b8d3a\") "
Sep 07 10:54:08.183341 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.183318    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b9b65b3-21e2-47c0-9dac-b876598b8d3a" (UID: "9b9b65b3-21e2-47c0-9dac-b876598b8d3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:54:08.185145 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.185120    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df" (OuterVolumeSpecName: "kube-api-access-f65df") pod "9b9b65b3-21e2-47c0-9dac-b876598b8d3a" (UID: "9b9b65b3-21e2-47c0-9dac-b876598b8d3a"). InnerVolumeSpecName "kube-api-access-f65df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:54:08.283883 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.283857    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f65df\" (UniqueName: \"kubernetes.io/projected/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-kube-api-access-f65df\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:08.283883 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.283879    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b9b65b3-21e2-47c0-9dac-b876598b8d3a-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:08.766633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.766552    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" containerID="8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7" exitCode=0
Sep 07 10:54:08.766633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.766600    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"
Sep 07 10:54:08.766633 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.766612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" event={"ID":"9b9b65b3-21e2-47c0-9dac-b876598b8d3a","Type":"ContainerDied","Data":"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"}
Sep 07 10:54:08.766867 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.766660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl" event={"ID":"9b9b65b3-21e2-47c0-9dac-b876598b8d3a","Type":"ContainerDied","Data":"3e5b9d9637bbad8e6cfb772deefcf08623cdafbf8e02c9b1717ffe6e526156e6"}
Sep 07 10:54:08.766867 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.766683    2562 scope.go:117] "RemoveContainer" containerID="8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"
Sep 07 10:54:08.774648 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.774632    2562 scope.go:117] "RemoveContainer" containerID="8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"
Sep 07 10:54:08.774925 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:54:08.774903    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7\": container with ID starting with 8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7 not found: ID does not exist" containerID="8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"
Sep 07 10:54:08.774970 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.774938    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7"} err="failed to get container status \"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7\": rpc error: code = NotFound desc = could not find container \"8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7\": container with ID starting with 8d24f46fc0fc5c8f44f15d24c1e83326533671994d3eeb4e143eb626249c2ea7 not found: ID does not exist"
Sep 07 10:54:08.785083 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.785062    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:54:08.788170 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:08.788146    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bef02f-6f6d657cf4-w5sgl"]
Sep 07 10:54:10.495342 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:10.495292    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" path="/var/lib/kubelet/pods/9b9b65b3-21e2-47c0-9dac-b876598b8d3a/volumes"
Sep 07 10:54:13.359478 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:13.359441    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:54:28.378811 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.378763    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerName="main" containerID="cri-o://66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a" gracePeriod=45
Sep 07 10:54:28.395691 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.395661    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 07 10:54:28.628549 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.628524    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:54:28.745029 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.744950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745029 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.744991    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745029 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745007    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745335 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745044    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745335 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5znsw\" (UniqueName: \"kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745335 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir\") pod \"af945369-7c41-4fd5-8d46-e83864f8efa6\" (UID: \"af945369-7c41-4fd5-8d46-e83864f8efa6\") "
Sep 07 10:54:28.745491 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745357    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache" (OuterVolumeSpecName: "model-cache") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:54:28.745491 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home" (OuterVolumeSpecName: "home") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:54:28.745575 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.745560    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:54:28.747421 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.747403    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw" (OuterVolumeSpecName: "kube-api-access-5znsw") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "kube-api-access-5znsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:54:28.747566 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.747534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm" (OuterVolumeSpecName: "dshm") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:54:28.747682 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.747572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af945369-7c41-4fd5-8d46-e83864f8efa6" (UID: "af945369-7c41-4fd5-8d46-e83864f8efa6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:54:28.845837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845811    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.845837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845834    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af945369-7c41-4fd5-8d46-e83864f8efa6-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.845837 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845843    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.846037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845850    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.846037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845858    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af945369-7c41-4fd5-8d46-e83864f8efa6-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.846037 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.845866    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znsw\" (UniqueName: \"kubernetes.io/projected/af945369-7c41-4fd5-8d46-e83864f8efa6-kube-api-access-5znsw\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:54:28.863234 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.863202    2562 generic.go:356] "Generic (PLEG): container finished" podID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerID="66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a" exitCode=0
Sep 07 10:54:28.863406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.863258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" event={"ID":"af945369-7c41-4fd5-8d46-e83864f8efa6","Type":"ContainerDied","Data":"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"}
Sep 07 10:54:28.863406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.863319    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t" event={"ID":"af945369-7c41-4fd5-8d46-e83864f8efa6","Type":"ContainerDied","Data":"79b5ddb665a64ce60a3d5db270d64ba45efae6b7fa20e4dc61962e96dfc9cc72"}
Sep 07 10:54:28.863406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.863266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"
Sep 07 10:54:28.863406 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.863337    2562 scope.go:117] "RemoveContainer" containerID="66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"
Sep 07 10:54:28.872378 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.872359    2562 scope.go:117] "RemoveContainer" containerID="66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"
Sep 07 10:54:28.872649 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:54:28.872631    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a\": container with ID starting with 66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a not found: ID does not exist" containerID="66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"
Sep 07 10:54:28.872700 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.872657    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a"} err="failed to get container status \"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a\": rpc error: code = NotFound desc = could not find container \"66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a\": container with ID starting with 66c80979afbcd767dbeaad072cf7be925a51103ad3ee5e6984b316f560dc165a not found: ID does not exist"
Sep 07 10:54:28.886676 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.886650    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:54:28.890551 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:28.890531    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bef02f/ipp-route-bef02f-kserve-6555cb6bd7-4zx2t"]
Sep 07 10:54:30.497004 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:54:30.496963    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" path="/var/lib/kubelet/pods/af945369-7c41-4fd5-8d46-e83864f8efa6/volumes"
Sep 07 10:55:07.551706 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.551670    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:55:07.552124 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.551959    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" podUID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" containerName="payload-processing" containerID="cri-o://33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98" gracePeriod=30
Sep 07 10:55:07.806043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.805979    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:55:07.877715 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.877679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume\") pod \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") "
Sep 07 10:55:07.877901 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.877805    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr5z5\" (UniqueName: \"kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5\") pod \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\" (UID: \"89a2324b-cadd-4c76-a5a0-70a35fa40ec9\") "
Sep 07 10:55:07.878039 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.878014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89a2324b-cadd-4c76-a5a0-70a35fa40ec9" (UID: "89a2324b-cadd-4c76-a5a0-70a35fa40ec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:55:07.879974 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.879948    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5" (OuterVolumeSpecName: "kube-api-access-zr5z5") pod "89a2324b-cadd-4c76-a5a0-70a35fa40ec9" (UID: "89a2324b-cadd-4c76-a5a0-70a35fa40ec9"). InnerVolumeSpecName "kube-api-access-zr5z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:55:07.978689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.978652    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr5z5\" (UniqueName: \"kubernetes.io/projected/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-kube-api-access-zr5z5\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:07.978689 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:07.978683    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a2324b-cadd-4c76-a5a0-70a35fa40ec9-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:08.036967 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.036932    2562 generic.go:356] "Generic (PLEG): container finished" podID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" containerID="33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98" exitCode=0
Sep 07 10:55:08.037155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.036986    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"
Sep 07 10:55:08.037155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.037021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" event={"ID":"89a2324b-cadd-4c76-a5a0-70a35fa40ec9","Type":"ContainerDied","Data":"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"}
Sep 07 10:55:08.037155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.037077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm" event={"ID":"89a2324b-cadd-4c76-a5a0-70a35fa40ec9","Type":"ContainerDied","Data":"1b80d71babaf2ea996303a128c6d6b1d3627aa87796e58ff2ebd311e7482ed6e"}
Sep 07 10:55:08.037155 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.037100    2562 scope.go:117] "RemoveContainer" containerID="33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"
Sep 07 10:55:08.045735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.045718    2562 scope.go:117] "RemoveContainer" containerID="33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"
Sep 07 10:55:08.046019 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:55:08.045998    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98\": container with ID starting with 33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98 not found: ID does not exist" containerID="33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"
Sep 07 10:55:08.046070 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.046029    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98"} err="failed to get container status \"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98\": rpc error: code = NotFound desc = could not find container \"33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98\": container with ID starting with 33541df153859a205a1cab47daaf8f6bbedba884632076aeb2dc078cfa14ca98 not found: ID does not exist"
Sep 07 10:55:08.058621 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.058568    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:55:08.062735 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.062714    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-738923-644fbcfb59-xlwzm"]
Sep 07 10:55:08.496356 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:08.496325    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" path="/var/lib/kubelet/pods/89a2324b-cadd-4c76-a5a0-70a35fa40ec9/volumes"
Sep 07 10:55:12.556511 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:12.556479    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:55:12.559414 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:12.559394    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 10:55:14.355579 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:14.355547    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:55:14.356063 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:14.355758    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy" containerID="cri-o://91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d" gracePeriod=30
Sep 07 10:55:19.608284 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.608257    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:55:19.677924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.677924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677829    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.677924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.677924 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.677968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678012    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whn2l\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l\") pod \"b243a441-6c46-4126-b3c8-93c698466b12\" (UID: \"b243a441-6c46-4126-b3c8-93c698466b12\") "
Sep 07 10:55:19.678253 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data" (OuterVolumeSpecName: "istio-data") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678445    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678504    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-data\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.678606 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.678541    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:55:19.680127 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.680104    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:55:19.680216 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.680192    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 10:55:19.680658 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.680630    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token" (OuterVolumeSpecName: "istio-token") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:55:19.680774 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.680757    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l" (OuterVolumeSpecName: "kube-api-access-whn2l") pod "b243a441-6c46-4126-b3c8-93c698466b12" (UID: "b243a441-6c46-4126-b3c8-93c698466b12"). InnerVolumeSpecName "kube-api-access-whn2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:55:19.779149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779109    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-credential-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779141    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b243a441-6c46-4126-b3c8-93c698466b12-istio-podinfo\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779151    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-istio-token\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779149 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779159    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-workload-socket\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779168    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b243a441-6c46-4126-b3c8-93c698466b12-istiod-ca-cert\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779178    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whn2l\" (UniqueName: \"kubernetes.io/projected/b243a441-6c46-4126-b3c8-93c698466b12-kube-api-access-whn2l\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:19.779459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:19.779186    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b243a441-6c46-4126-b3c8-93c698466b12-istio-envoy\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:20.092096 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.092064    2562 generic.go:356] "Generic (PLEG): container finished" podID="b243a441-6c46-4126-b3c8-93c698466b12" containerID="91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d" exitCode=0
Sep 07 10:55:20.092257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.092116    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"
Sep 07 10:55:20.092257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.092125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerDied","Data":"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"}
Sep 07 10:55:20.092257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.092178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh" event={"ID":"b243a441-6c46-4126-b3c8-93c698466b12","Type":"ContainerDied","Data":"a95ad8992494b06211898696c2042c6fe442b82cd327cca0a0ba3bbe67d38972"}
Sep 07 10:55:20.092257 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.092208    2562 scope.go:117] "RemoveContainer" containerID="91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"
Sep 07 10:55:20.101271 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.101237    2562 scope.go:117] "RemoveContainer" containerID="35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"
Sep 07 10:55:20.109589 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.109570    2562 scope.go:117] "RemoveContainer" containerID="91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"
Sep 07 10:55:20.109886 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:55:20.109866    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d\": container with ID starting with 91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d not found: ID does not exist" containerID="91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"
Sep 07 10:55:20.109942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.109896    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d"} err="failed to get container status \"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d\": rpc error: code = NotFound desc = could not find container \"91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d\": container with ID starting with 91af55943733d3e7d25203f75341acc23094812e8d44897fe0805e9575d7126d not found: ID does not exist"
Sep 07 10:55:20.109942 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.109913    2562 scope.go:117] "RemoveContainer" containerID="35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"
Sep 07 10:55:20.110172 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:55:20.110153    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2\": container with ID starting with 35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2 not found: ID does not exist" containerID="35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"
Sep 07 10:55:20.110212 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.110180    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2"} err="failed to get container status \"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2\": rpc error: code = NotFound desc = could not find container \"35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2\": container with ID starting with 35f40fd2ea3780df607c9df5cb2c1a2976db5e64cb6d7d50918a3988331074a2 not found: ID does not exist"
Sep 07 10:55:20.112859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.112838    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:55:20.117420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.117397    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-738923-openshift-default-86dc499d64-v84mh"]
Sep 07 10:55:20.495459 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:20.495378    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b243a441-6c46-4126-b3c8-93c698466b12" path="/var/lib/kubelet/pods/b243a441-6c46-4126-b3c8-93c698466b12/volumes"
Sep 07 10:55:25.329960 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.329923    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:55:25.330441 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.330158    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" podUID="5c55085a-9da3-4276-94ac-2542b3c5c39e" containerName="payload-processing" containerID="cri-o://b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78" gracePeriod=30
Sep 07 10:55:25.339003 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.338973    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:55:25.339173 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.339131    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" podUID="6dd89136-5260-4d69-9621-482f55b85a9b" containerName="payload-pre-processing" containerID="cri-o://a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a" gracePeriod=30
Sep 07 10:55:25.605043 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.605013    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:55:25.607859 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.607838    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:55:25.732487 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732454    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume\") pod \"6dd89136-5260-4d69-9621-482f55b85a9b\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") "
Sep 07 10:55:25.732670 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732506    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w78xf\" (UniqueName: \"kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf\") pod \"5c55085a-9da3-4276-94ac-2542b3c5c39e\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") "
Sep 07 10:55:25.732670 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732522    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume\") pod \"5c55085a-9da3-4276-94ac-2542b3c5c39e\" (UID: \"5c55085a-9da3-4276-94ac-2542b3c5c39e\") "
Sep 07 10:55:25.732670 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kj2g\" (UniqueName: \"kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g\") pod \"6dd89136-5260-4d69-9621-482f55b85a9b\" (UID: \"6dd89136-5260-4d69-9621-482f55b85a9b\") "
Sep 07 10:55:25.732855 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732839    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c55085a-9da3-4276-94ac-2542b3c5c39e" (UID: "5c55085a-9da3-4276-94ac-2542b3c5c39e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:55:25.732933 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732898    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c55085a-9da3-4276-94ac-2542b3c5c39e-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:25.732933 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.732893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dd89136-5260-4d69-9621-482f55b85a9b" (UID: "6dd89136-5260-4d69-9621-482f55b85a9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:55:25.734843 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.734819    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g" (OuterVolumeSpecName: "kube-api-access-6kj2g") pod "6dd89136-5260-4d69-9621-482f55b85a9b" (UID: "6dd89136-5260-4d69-9621-482f55b85a9b"). InnerVolumeSpecName "kube-api-access-6kj2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:55:25.734940 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.734858    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf" (OuterVolumeSpecName: "kube-api-access-w78xf") pod "5c55085a-9da3-4276-94ac-2542b3c5c39e" (UID: "5c55085a-9da3-4276-94ac-2542b3c5c39e"). InnerVolumeSpecName "kube-api-access-w78xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:55:25.833613 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.833534    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kj2g\" (UniqueName: \"kubernetes.io/projected/6dd89136-5260-4d69-9621-482f55b85a9b-kube-api-access-6kj2g\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:25.833613 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.833563    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd89136-5260-4d69-9621-482f55b85a9b-plugins-config-volume\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:25.833613 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:25.833573    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w78xf\" (UniqueName: \"kubernetes.io/projected/5c55085a-9da3-4276-94ac-2542b3c5c39e-kube-api-access-w78xf\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:26.124931 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.124835    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c55085a-9da3-4276-94ac-2542b3c5c39e" containerID="b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78" exitCode=0
Sep 07 10:55:26.124931 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.124877    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"
Sep 07 10:55:26.125158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.124923    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" event={"ID":"5c55085a-9da3-4276-94ac-2542b3c5c39e","Type":"ContainerDied","Data":"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"}
Sep 07 10:55:26.125158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.124973    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6" event={"ID":"5c55085a-9da3-4276-94ac-2542b3c5c39e","Type":"ContainerDied","Data":"b4c0ea48d14aa8c18267f379786c46040c333e3294236f1602722576c5128b8d"}
Sep 07 10:55:26.125158 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.124996    2562 scope.go:117] "RemoveContainer" containerID="b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"
Sep 07 10:55:26.126269 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.126245    2562 generic.go:356] "Generic (PLEG): container finished" podID="6dd89136-5260-4d69-9621-482f55b85a9b" containerID="a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a" exitCode=0
Sep 07 10:55:26.126392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.126288    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"
Sep 07 10:55:26.126392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.126294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" event={"ID":"6dd89136-5260-4d69-9621-482f55b85a9b","Type":"ContainerDied","Data":"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"}
Sep 07 10:55:26.126392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.126350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml" event={"ID":"6dd89136-5260-4d69-9621-482f55b85a9b","Type":"ContainerDied","Data":"fad47d4b9fbc6c7773c975caa581cab040d2854066478cffe1bf4447f9940e24"}
Sep 07 10:55:26.133845 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.133828    2562 scope.go:117] "RemoveContainer" containerID="b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"
Sep 07 10:55:26.134249 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:55:26.134115    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78\": container with ID starting with b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78 not found: ID does not exist" containerID="b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"
Sep 07 10:55:26.134249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.134142    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78"} err="failed to get container status \"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78\": rpc error: code = NotFound desc = could not find container \"b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78\": container with ID starting with b76848e24fe86692a71691b92cc490d99b55ce3da5c3ce5b089eaff035bd9d78 not found: ID does not exist"
Sep 07 10:55:26.134249 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.134166    2562 scope.go:117] "RemoveContainer" containerID="a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"
Sep 07 10:55:26.141719 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.141702    2562 scope.go:117] "RemoveContainer" containerID="a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"
Sep 07 10:55:26.141976 ip-10-0-142-36 kubenswrapper[2562]: E0907 10:55:26.141958    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a\": container with ID starting with a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a not found: ID does not exist" containerID="a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"
Sep 07 10:55:26.142040 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.141980    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a"} err="failed to get container status \"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a\": rpc error: code = NotFound desc = could not find container \"a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a\": container with ID starting with a8db16c444dcc31a43aec40351551006d57f5c0cf18ab96106710014bb5ecd7a not found: ID does not exist"
Sep 07 10:55:26.150217 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.150196    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:55:26.154397 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.154377    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-5a0ff8-59bdbbbc75-9mhw6"]
Sep 07 10:55:26.164806 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.164788    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:55:26.169787 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.169767    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-5a0ff8-7d7484d94-qdnml"]
Sep 07 10:55:26.495886 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.495812    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c55085a-9da3-4276-94ac-2542b3c5c39e" path="/var/lib/kubelet/pods/5c55085a-9da3-4276-94ac-2542b3c5c39e/volumes"
Sep 07 10:55:26.496223 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:26.496142    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd89136-5260-4d69-9621-482f55b85a9b" path="/var/lib/kubelet/pods/6dd89136-5260-4d69-9621-482f55b85a9b/volumes"
Sep 07 10:55:32.647995 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.647962    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"]
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648235    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" containerName="payload-pre-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648247    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" containerName="payload-pre-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648255    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648260    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648270    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648285    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" containerName="payload-pre-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648290    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" containerName="payload-pre-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648314    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerName="main"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648319    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerName="main"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648326    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" containerName="revoke-keys"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648332    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" containerName="revoke-keys"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648342    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" containerName="payload-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648347    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" containerName="payload-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648357    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c55085a-9da3-4276-94ac-2542b3c5c39e" containerName="payload-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648363    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c55085a-9da3-4276-94ac-2542b3c5c39e" containerName="payload-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648368    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd89136-5260-4d69-9621-482f55b85a9b" containerName="payload-pre-processing"
Sep 07 10:55:32.648385 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648374    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd89136-5260-4d69-9621-482f55b85a9b" containerName="payload-pre-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648419    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="89a2324b-cadd-4c76-a5a0-70a35fa40ec9" containerName="payload-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648427    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ffcb36-c388-4404-9f3e-db0bcb475b8e" containerName="payload-pre-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648432    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5fd56c-3d97-4c2a-ac90-1a435c6d8827" containerName="revoke-keys"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648437    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd89136-5260-4d69-9621-482f55b85a9b" containerName="payload-pre-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648443    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b9b65b3-21e2-47c0-9dac-b876598b8d3a" containerName="payload-pre-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648447    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c55085a-9da3-4276-94ac-2542b3c5c39e" containerName="payload-processing"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648454    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648459    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b243a441-6c46-4126-b3c8-93c698466b12" containerName="istio-proxy"
Sep 07 10:55:32.648935 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.648464    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="af945369-7c41-4fd5-8d46-e83864f8efa6" containerName="main"
Sep 07 10:55:32.652843 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.652826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:32.655340 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.655318    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\""
Sep 07 10:55:32.659461 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.659437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"]
Sep 07 10:55:32.787392 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.787366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd\" (UID: \"f038ce22-999f-4ada-bbfa-79feb84fbb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:32.888607 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.888577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd\" (UID: \"f038ce22-999f-4ada-bbfa-79feb84fbb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:32.889154 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.889136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd\" (UID: \"f038ce22-999f-4ada-bbfa-79feb84fbb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:32.963945 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:32.963872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:33.233349 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:33.233262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"]
Sep 07 10:55:33.235913 ip-10-0-142-36 kubenswrapper[2562]: W0907 10:55:33.235891    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf038ce22_999f_4ada_bbfa_79feb84fbb24.slice/crio-3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe WatchSource:0}: Error finding container 3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe: Status 404 returned error can't find the container with id 3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe
Sep 07 10:55:34.163818 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:34.163727    2562 generic.go:356] "Generic (PLEG): container finished" podID="f038ce22-999f-4ada-bbfa-79feb84fbb24" containerID="27de3197b65e334d7d667f301554a13fbbe7a88b0b94b4e18ac0bf1b11b8a5e2" exitCode=0
Sep 07 10:55:34.164195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:34.163819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd" event={"ID":"f038ce22-999f-4ada-bbfa-79feb84fbb24","Type":"ContainerDied","Data":"27de3197b65e334d7d667f301554a13fbbe7a88b0b94b4e18ac0bf1b11b8a5e2"}
Sep 07 10:55:34.164195 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:34.163914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd" event={"ID":"f038ce22-999f-4ada-bbfa-79feb84fbb24","Type":"ContainerStarted","Data":"3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe"}
Sep 07 10:55:35.290420 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:35.290393    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:35.407535 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:35.407503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca\") pod \"f038ce22-999f-4ada-bbfa-79feb84fbb24\" (UID: \"f038ce22-999f-4ada-bbfa-79feb84fbb24\") "
Sep 07 10:55:35.407884 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:35.407859    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f038ce22-999f-4ada-bbfa-79feb84fbb24" (UID: "f038ce22-999f-4ada-bbfa-79feb84fbb24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:55:35.508337 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:35.508248    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f038ce22-999f-4ada-bbfa-79feb84fbb24-maas-api-service-ca\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 10:55:36.172967 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:36.172931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd" event={"ID":"f038ce22-999f-4ada-bbfa-79feb84fbb24","Type":"ContainerDied","Data":"3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe"}
Sep 07 10:55:36.172967 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:36.172963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"
Sep 07 10:55:36.173165 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:36.172981    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3df0bd27c9f8594810d2f86888579c320d88ce2dc46908ba58b3e3636bedc7fe"
Sep 07 10:55:43.219172 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:43.219137    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"]
Sep 07 10:55:43.221026 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:43.220999    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-659410-29812965-cjq6g"]
Sep 07 10:55:43.409532 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:43.409500    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"]
Sep 07 10:55:43.412055 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:43.412030    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-659410-s9qhd"]
Sep 07 10:55:44.494956 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:44.494923    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3599b21c-2b0e-46b7-b850-49022fa943e6" path="/var/lib/kubelet/pods/3599b21c-2b0e-46b7-b850-49022fa943e6/volumes"
Sep 07 10:55:44.495346 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:55:44.495200    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f038ce22-999f-4ada-bbfa-79feb84fbb24" path="/var/lib/kubelet/pods/f038ce22-999f-4ada-bbfa-79feb84fbb24/volumes"
Sep 07 10:56:12.749897 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:56:12.749862    2562 scope.go:117] "RemoveContainer" containerID="aa7c7b1558d6c09c2d984d69da2d785fe61e1c1b584ddf884b77311e1df58fd1"
Sep 07 10:56:12.757758 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:56:12.757736    2562 scope.go:117] "RemoveContainer" containerID="210ab1e5c8b9a4cfb26e4d0e2d5c4f9fffa51a85c71be119740158e9adf25eb3"
Sep 07 10:56:12.764846 ip-10-0-142-36 kubenswrapper[2562]: I0907 10:56:12.764831    2562 scope.go:117] "RemoveContainer" containerID="f59efa9fe08baaf47adb8b03a105d5ae1214bb741b58dd131a79b431cada5801"
Sep 07 11:00:06.325645 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.325609    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:00:06.326085 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.326034    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f038ce22-999f-4ada-bbfa-79feb84fbb24" containerName="revoke-keys"
Sep 07 11:00:06.326085 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.326049    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f038ce22-999f-4ada-bbfa-79feb84fbb24" containerName="revoke-keys"
Sep 07 11:00:06.326183 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.326134    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f038ce22-999f-4ada-bbfa-79feb84fbb24" containerName="revoke-keys"
Sep 07 11:00:06.328934 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.328911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.331424 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.331400    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-434fae01-kserve-self-signed-certs\""
Sep 07 11:00:06.338949 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.338930    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:00:06.429556 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.429714 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.429714 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbk2\" (UniqueName: \"kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.429714 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429634    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.429714 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.429980 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.429740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530185 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530380 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530380 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530380 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530380 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbk2\" (UniqueName: \"kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530380 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530634 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530634 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.530771 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.530748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.532692 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.532665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.532938 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.532920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.538880 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.538857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbk2\" (UniqueName: \"kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2\") pod \"e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") " pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.638656 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.638582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:06.911180 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:06.911154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:00:06.913579 ip-10-0-142-36 kubenswrapper[2562]: W0907 11:00:06.913549    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3032bd4e_9fab_4a79_a8bb_38c537bf87ac.slice/crio-c6d0bf15f348dc24c7067def460be0dc786b443ff525a4779856cb4ca544a163 WatchSource:0}: Error finding container c6d0bf15f348dc24c7067def460be0dc786b443ff525a4779856cb4ca544a163: Status 404 returned error can't find the container with id c6d0bf15f348dc24c7067def460be0dc786b443ff525a4779856cb4ca544a163
Sep 07 11:00:07.280933 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:07.280840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" event={"ID":"3032bd4e-9fab-4a79-a8bb-38c537bf87ac","Type":"ContainerStarted","Data":"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"}
Sep 07 11:00:07.280933 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:07.280895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" event={"ID":"3032bd4e-9fab-4a79-a8bb-38c537bf87ac","Type":"ContainerStarted","Data":"c6d0bf15f348dc24c7067def460be0dc786b443ff525a4779856cb4ca544a163"}
Sep 07 11:00:07.300836 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:07.300790    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" podStartSLOduration=1.300776372 podStartE2EDuration="1.300776372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:07.299120505 +0000 UTC m=+2395.309773831" watchObservedRunningTime="2026-09-07 11:00:07.300776372 +0000 UTC m=+2395.311429687"
Sep 07 11:00:12.584851 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:12.584821    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 11:00:12.590464 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:12.590437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k52cb_1bdccc36-a482-400f-85d1-f9fcdbe5a277/ovn-acl-logging/0.log"
Sep 07 11:00:12.817706 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:12.817678    2562 scope.go:117] "RemoveContainer" containerID="a1239b5dba0ef7576e34c9c1faf5291439d87ee47cb29b5647e8edffc2157fe6"
Sep 07 11:00:16.639273 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:16.639242    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:16.639683 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:16.639293    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:16.651895 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:16.651871    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:17.333613 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:17.333581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:00:21.132142 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.132098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:00:21.135317 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.135277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.137883 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.137863    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-434fae01-kserve-self-signed-certs\""
Sep 07 11:00:21.150270 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.150242    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:00:21.161180 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161156    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69ww\" (UniqueName: \"kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.161351 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.161351 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.161351 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.161489 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161357    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.161489 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.161385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.261947 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.261902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69ww\" (UniqueName: \"kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262157 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.261956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262157 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.261995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262157 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262157 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262157 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262063    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262480 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262540 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262507    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.262582 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.262543    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.264567 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.264540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.264722 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.264705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.269767 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.269745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69ww\" (UniqueName: \"kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww\") pod \"e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") " pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.444279 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.444178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:21.715426 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:21.715341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:00:21.718267 ip-10-0-142-36 kubenswrapper[2562]: W0907 11:00:21.718233    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd34acaf0_0159_4631_9197_17ead5b988e0.slice/crio-d75138040b642f2549dc400e5e6f4bfd71776796d591974fe8e6f6296e18aefd WatchSource:0}: Error finding container d75138040b642f2549dc400e5e6f4bfd71776796d591974fe8e6f6296e18aefd: Status 404 returned error can't find the container with id d75138040b642f2549dc400e5e6f4bfd71776796d591974fe8e6f6296e18aefd
Sep 07 11:00:22.346074 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:22.346037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" event={"ID":"d34acaf0-0159-4631-9197-17ead5b988e0","Type":"ContainerStarted","Data":"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"}
Sep 07 11:00:22.346514 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:22.346081    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" event={"ID":"d34acaf0-0159-4631-9197-17ead5b988e0","Type":"ContainerStarted","Data":"d75138040b642f2549dc400e5e6f4bfd71776796d591974fe8e6f6296e18aefd"}
Sep 07 11:00:22.364040 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:22.363991    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" podStartSLOduration=1.363977855 podStartE2EDuration="1.363977855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:22.362203407 +0000 UTC m=+2410.372856712" watchObservedRunningTime="2026-09-07 11:00:22.363977855 +0000 UTC m=+2410.374631199"
Sep 07 11:00:31.445135 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:31.445093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:31.445602 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:31.445155    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:31.457661 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:31.457634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:00:32.398833 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:00:32.398799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:01:07.770004 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:07.769964    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:01:09.472730 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:09.472691    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:01:22.790192 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:22.790154    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" podUID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" containerName="main" containerID="cri-o://c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845" gracePeriod=45
Sep 07 11:01:23.035627 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.035601    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:01:23.107058 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107032    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107229 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107229 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107123    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107229 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107151    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107229 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107229 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sbk2\" (UniqueName: \"kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2\") pod \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\" (UID: \"3032bd4e-9fab-4a79-a8bb-38c537bf87ac\") "
Sep 07 11:01:23.107517 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107443    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:23.107517 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107455    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home" (OuterVolumeSpecName: "home") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:23.107517 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.107469    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:23.109391 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.109366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm" (OuterVolumeSpecName: "dshm") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:23.109503 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.109380    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2" (OuterVolumeSpecName: "kube-api-access-7sbk2") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "kube-api-access-7sbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:01:23.109503 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.109402    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3032bd4e-9fab-4a79-a8bb-38c537bf87ac" (UID: "3032bd4e-9fab-4a79-a8bb-38c537bf87ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:01:23.207865 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207833    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.207865 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207858    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.207865 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207867    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.208063 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207875    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.208063 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207882    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.208063 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.207891    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbk2\" (UniqueName: \"kubernetes.io/projected/3032bd4e-9fab-4a79-a8bb-38c537bf87ac-kube-api-access-7sbk2\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:23.595771 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.595739    2562 generic.go:356] "Generic (PLEG): container finished" podID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" containerID="c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845" exitCode=0
Sep 07 11:01:23.596015 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.595781    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"
Sep 07 11:01:23.596015 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.595781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" event={"ID":"3032bd4e-9fab-4a79-a8bb-38c537bf87ac","Type":"ContainerDied","Data":"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"}
Sep 07 11:01:23.596015 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.595837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k" event={"ID":"3032bd4e-9fab-4a79-a8bb-38c537bf87ac","Type":"ContainerDied","Data":"c6d0bf15f348dc24c7067def460be0dc786b443ff525a4779856cb4ca544a163"}
Sep 07 11:01:23.596015 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.595861    2562 scope.go:117] "RemoveContainer" containerID="c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"
Sep 07 11:01:23.604097 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.604083    2562 scope.go:117] "RemoveContainer" containerID="c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"
Sep 07 11:01:23.604400 ip-10-0-142-36 kubenswrapper[2562]: E0907 11:01:23.604379    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845\": container with ID starting with c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845 not found: ID does not exist" containerID="c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"
Sep 07 11:01:23.604447 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.604409    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845"} err="failed to get container status \"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845\": rpc error: code = NotFound desc = could not find container \"c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845\": container with ID starting with c99a5790823e2a6d940f86bff84656a0321b91e6f5be5cf2f1c8920733b7a845 not found: ID does not exist"
Sep 07 11:01:23.618663 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.618640    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:01:23.622230 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:23.622207    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-434fae01-kserve-58b5447599-wmx6k"]
Sep 07 11:01:24.493073 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.493035    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" podUID="d34acaf0-0159-4631-9197-17ead5b988e0" containerName="main" containerID="cri-o://2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8" gracePeriod=45
Sep 07 11:01:24.495472 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.495446    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" path="/var/lib/kubelet/pods/3032bd4e-9fab-4a79-a8bb-38c537bf87ac/volumes"
Sep 07 11:01:24.738121 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.738088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:01:24.822151 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822284 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822163    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822284 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j69ww\" (UniqueName: \"kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822284 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822269    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822505 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822505 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir\") pod \"d34acaf0-0159-4631-9197-17ead5b988e0\" (UID: \"d34acaf0-0159-4631-9197-17ead5b988e0\") "
Sep 07 11:01:24.822505 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822462    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home" (OuterVolumeSpecName: "home") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:24.822659 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822541    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:24.822724 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822708    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-model-cache\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:24.822773 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:24.822773 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.822729    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-home\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:24.824364 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.824341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 11:01:24.824974 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.824949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm" (OuterVolumeSpecName: "dshm") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 11:01:24.824974 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.824952    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww" (OuterVolumeSpecName: "kube-api-access-j69ww") pod "d34acaf0-0159-4631-9197-17ead5b988e0" (UID: "d34acaf0-0159-4631-9197-17ead5b988e0"). InnerVolumeSpecName "kube-api-access-j69ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 11:01:24.924060 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.924034    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34acaf0-0159-4631-9197-17ead5b988e0-tls-certs\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:24.924060 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.924056    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69ww\" (UniqueName: \"kubernetes.io/projected/d34acaf0-0159-4631-9197-17ead5b988e0-kube-api-access-j69ww\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:24.924202 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.924069    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-dshm\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:24.924202 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:24.924083    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d34acaf0-0159-4631-9197-17ead5b988e0-tmp-dir\") on node \"ip-10-0-142-36.ec2.internal\" DevicePath \"\""
Sep 07 11:01:25.605489 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.605455    2562 generic.go:356] "Generic (PLEG): container finished" podID="d34acaf0-0159-4631-9197-17ead5b988e0" containerID="2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8" exitCode=0
Sep 07 11:01:25.605905 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.605501    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"
Sep 07 11:01:25.605905 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.605546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" event={"ID":"d34acaf0-0159-4631-9197-17ead5b988e0","Type":"ContainerDied","Data":"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"}
Sep 07 11:01:25.605905 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.605584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf" event={"ID":"d34acaf0-0159-4631-9197-17ead5b988e0","Type":"ContainerDied","Data":"d75138040b642f2549dc400e5e6f4bfd71776796d591974fe8e6f6296e18aefd"}
Sep 07 11:01:25.605905 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.605603    2562 scope.go:117] "RemoveContainer" containerID="2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"
Sep 07 11:01:25.614258 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.614241    2562 scope.go:117] "RemoveContainer" containerID="2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"
Sep 07 11:01:25.614527 ip-10-0-142-36 kubenswrapper[2562]: E0907 11:01:25.614511    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8\": container with ID starting with 2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8 not found: ID does not exist" containerID="2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"
Sep 07 11:01:25.614583 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.614535    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8"} err="failed to get container status \"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8\": rpc error: code = NotFound desc = could not find container \"2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8\": container with ID starting with 2da37d85d785f5c3d41e864685f13d0abd1ccf4c4b30e40a2ed1c335b73e98b8 not found: ID does not exist"
Sep 07 11:01:25.626667 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.626642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:01:25.631068 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:25.631045    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-434fae01-kserve-675f587b55-hx6rf"]
Sep 07 11:01:26.496448 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:01:26.496414    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34acaf0-0159-4631-9197-17ead5b988e0" path="/var/lib/kubelet/pods/d34acaf0-0159-4631-9197-17ead5b988e0/volumes"
Sep 07 11:02:06.851331 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:06.851286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56d56b9c8c-txqjs_3c5f1dbd-4ffc-45f3-8522-906a0ef963d9/authorino/0.log"
Sep 07 11:02:12.841719 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:12.841689    2562 scope.go:117] "RemoveContainer" containerID="27de3197b65e334d7d667f301554a13fbbe7a88b0b94b4e18ac0bf1b11b8a5e2"
Sep 07 11:02:14.720793 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:14.720762    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-6tv7t_426f9824-2a3f-4118-9d9f-4c7852c8462f/copy-manifests/0.log"
Sep 07 11:02:14.734899 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:14.734872    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-6tv7t_426f9824-2a3f-4118-9d9f-4c7852c8462f/manager/0.log"
Sep 07 11:02:15.974425 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:15.974395    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-22kwg_d9f78643-8675-42a2-8aa0-d7e4770ab1cf/postgres/0.log"
Sep 07 11:02:16.919407 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:16.919378    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56d56b9c8c-txqjs_3c5f1dbd-4ffc-45f3-8522-906a0ef963d9/authorino/0.log"
Sep 07 11:02:18.985655 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:18.985623    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7794764676-c68qd_c6c8671f-5ef3-421f-a99b-e4c02bf46504/kube-auth-proxy/0.log"
Sep 07 11:02:20.182442 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:20.182414    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6gwzv_e755632e-3f91-43c8-b295-43aa4756622a/main/0.log"
Sep 07 11:02:20.443268 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:20.443198    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-29v4f_e8740642-e00c-4e33-82ac-4ede3f52e6b3/main/0.log"
Sep 07 11:02:20.715587 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:20.715498    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-phn72_e3aed68f-f782-43f2-a816-5bfd4c498cd7/main/0.log"
Sep 07 11:02:20.854881 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:20.854853    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cxkw8_820292cb-5028-44e9-ab4b-ed5fe71e7fba/main/0.log"
Sep 07 11:02:30.382357 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:30.382323    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tc82s_d824971e-4bb2-4633-b3b6-f705d282054c/global-pull-secret-syncer/0.log"
Sep 07 11:02:30.475588 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:30.475562    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-84zcq_80ee1345-8c97-4dd4-8534-31c70c349b80/konnectivity-agent/0.log"
Sep 07 11:02:30.574883 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:30.574856    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-36.ec2.internal_e3f124c2f31ba99ab09255e938134ba7/haproxy/0.log"
Sep 07 11:02:35.406805 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:35.406773    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56d56b9c8c-txqjs_3c5f1dbd-4ffc-45f3-8522-906a0ef963d9/authorino/0.log"
Sep 07 11:02:37.480173 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:37.480146    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tgwg_ee3e0294-8df7-4731-ac8c-ef5c025134a4/node-exporter/0.log"
Sep 07 11:02:37.508174 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:37.508140    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tgwg_ee3e0294-8df7-4731-ac8c-ef5c025134a4/kube-rbac-proxy/0.log"
Sep 07 11:02:37.532071 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:37.532046    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5tgwg_ee3e0294-8df7-4731-ac8c-ef5c025134a4/init-textfile/0.log"
Sep 07 11:02:40.056867 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.056823    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"]
Sep 07 11:02:40.057393 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057235    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" containerName="main"
Sep 07 11:02:40.057393 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" containerName="main"
Sep 07 11:02:40.057393 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057293    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34acaf0-0159-4631-9197-17ead5b988e0" containerName="main"
Sep 07 11:02:40.057393 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057319    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34acaf0-0159-4631-9197-17ead5b988e0" containerName="main"
Sep 07 11:02:40.057393 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057395    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3032bd4e-9fab-4a79-a8bb-38c537bf87ac" containerName="main"
Sep 07 11:02:40.057670 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.057404    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34acaf0-0159-4631-9197-17ead5b988e0" containerName="main"
Sep 07 11:02:40.060509 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.060486    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.063410 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.063394    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxx4\"/\"openshift-service-ca.crt\""
Sep 07 11:02:40.063495 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.063418    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxx4\"/\"kube-root-ca.crt\""
Sep 07 11:02:40.064511 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.064492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jhxx4\"/\"default-dockercfg-jnqbs\""
Sep 07 11:02:40.070026 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.069996    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"]
Sep 07 11:02:40.200238 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.200181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-podres\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.200238 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.200241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dtl\" (UniqueName: \"kubernetes.io/projected/eb29eca5-9700-4f55-94ab-597ea3fd5b80-kube-api-access-b2dtl\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.200490 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.200354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-proc\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.200490 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.200389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-sys\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.200490 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.200407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-lib-modules\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301046 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.300998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-proc\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-sys\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-lib-modules\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-proc\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-podres\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-sys\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301212 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dtl\" (UniqueName: \"kubernetes.io/projected/eb29eca5-9700-4f55-94ab-597ea3fd5b80-kube-api-access-b2dtl\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301463 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-podres\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.301463 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.301265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb29eca5-9700-4f55-94ab-597ea3fd5b80-lib-modules\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.309523 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.309462    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dtl\" (UniqueName: \"kubernetes.io/projected/eb29eca5-9700-4f55-94ab-597ea3fd5b80-kube-api-access-b2dtl\") pod \"perf-node-gather-daemonset-bgcmt\" (UID: \"eb29eca5-9700-4f55-94ab-597ea3fd5b80\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.370913 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.370873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:40.638794 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.638769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"]
Sep 07 11:02:40.641106 ip-10-0-142-36 kubenswrapper[2562]: W0907 11:02:40.641071    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb29eca5_9700_4f55_94ab_597ea3fd5b80.slice/crio-b586b1bd6cf61029368e0cf848473bf3c2720711901a44ed4803b8dbf7f93d0d WatchSource:0}: Error finding container b586b1bd6cf61029368e0cf848473bf3c2720711901a44ed4803b8dbf7f93d0d: Status 404 returned error can't find the container with id b586b1bd6cf61029368e0cf848473bf3c2720711901a44ed4803b8dbf7f93d0d
Sep 07 11:02:40.910068 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.909982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt" event={"ID":"eb29eca5-9700-4f55-94ab-597ea3fd5b80","Type":"ContainerStarted","Data":"edea56562bcafb16125a33edc584cce82d4656ccd8b7fd6f68bcb74fcbb2d51c"}
Sep 07 11:02:40.910068 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.910033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt" event={"ID":"eb29eca5-9700-4f55-94ab-597ea3fd5b80","Type":"ContainerStarted","Data":"b586b1bd6cf61029368e0cf848473bf3c2720711901a44ed4803b8dbf7f93d0d"}
Sep 07 11:02:40.910266 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:40.910092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:42.761788 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:42.761764    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6m9dk_245a647f-a221-48f1-8175-37626932d896/dns/0.log"
Sep 07 11:02:42.786498 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:42.786462    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6m9dk_245a647f-a221-48f1-8175-37626932d896/kube-rbac-proxy/0.log"
Sep 07 11:02:42.924110 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:42.924081    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xl45g_777e3ff7-f089-4236-b771-d0d365587433/dns-node-resolver/0.log"
Sep 07 11:02:43.447008 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:43.446979    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b88cc7879-lhdgt_0e939e49-8cbe-46f9-8000-6c5d0b68f78c/registry/0.log"
Sep 07 11:02:43.468354 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:43.468325    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmtwk_034465eb-ecc3-4b74-836b-772dfde3553b/node-ca/0.log"
Sep 07 11:02:44.802648 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:44.802621    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7794764676-c68qd_c6c8671f-5ef3-421f-a99b-e4c02bf46504/kube-auth-proxy/0.log"
Sep 07 11:02:45.803828 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:45.803777    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hvdjj_5847744b-f489-40d7-a83e-04fa5553b601/serve-healthcheck-canary/0.log"
Sep 07 11:02:46.405500 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:46.405445    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sd5vw_9991cffd-bfa1-4c64-8e07-4800532d94e4/kube-rbac-proxy/0.log"
Sep 07 11:02:46.429050 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:46.429022    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sd5vw_9991cffd-bfa1-4c64-8e07-4800532d94e4/exporter/0.log"
Sep 07 11:02:46.451085 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:46.451050    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sd5vw_9991cffd-bfa1-4c64-8e07-4800532d94e4/extractor/0.log"
Sep 07 11:02:46.923636 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:46.923610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt"
Sep 07 11:02:46.945023 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:46.944975    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-bgcmt" podStartSLOduration=6.944960623 podStartE2EDuration="6.944960623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:40.936962013 +0000 UTC m=+2548.947615328" watchObservedRunningTime="2026-09-07 11:02:46.944960623 +0000 UTC m=+2554.955613938"
Sep 07 11:02:49.190718 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:49.190687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-6tv7t_426f9824-2a3f-4118-9d9f-4c7852c8462f/manager/0.log"
Sep 07 11:02:49.210272 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:49.210248    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-6tv7t_426f9824-2a3f-4118-9d9f-4c7852c8462f/copy-manifests/0.log"
Sep 07 11:02:50.670080 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:50.670058    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-547fb475d5-5z9lr_34b7c087-0faf-4a41-8b99-9c3935f2c8a6/manager/0.log"
Sep 07 11:02:51.105615 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:51.105585    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7794764676-c68qd_c6c8671f-5ef3-421f-a99b-e4c02bf46504/kube-auth-proxy/0.log"
Sep 07 11:02:53.039177 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:53.039099    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7794764676-c68qd_c6c8671f-5ef3-421f-a99b-e4c02bf46504/kube-auth-proxy/0.log"
Sep 07 11:02:57.228411 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.228383    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6jvg7_a17b86d7-2d73-4a68-b930-2c58ec10967a/kube-multus/0.log"
Sep 07 11:02:57.597708 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.597681    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/kube-multus-additional-cni-plugins/0.log"
Sep 07 11:02:57.620681 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.620634    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/egress-router-binary-copy/0.log"
Sep 07 11:02:57.644623 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.644603    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/cni-plugins/0.log"
Sep 07 11:02:57.668876 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.668851    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/bond-cni-plugin/0.log"
Sep 07 11:02:57.693394 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.693371    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/routeoverride-cni/0.log"
Sep 07 11:02:57.718995 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.718977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/whereabouts-cni-bincopy/0.log"
Sep 07 11:02:57.748188 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.748157    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tbw29_f7d31faf-f1f3-497a-a235-d1c7dd4fcdf9/whereabouts-cni/0.log"
Sep 07 11:02:57.986095 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:57.986025    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zhvfh_3e92996d-bebd-40a6-9f37-771ea7981a16/network-metrics-daemon/0.log"
Sep 07 11:02:58.006131 ip-10-0-142-36 kubenswrapper[2562]: I0907 11:02:58.006112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zhvfh_3e92996d-bebd-40a6-9f37-771ea7981a16/kube-rbac-proxy/0.log"