Sep 10 20:57:51.981100 ip-10-0-135-61 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:57:52.324800 ip-10-0-135-61 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:52.324800 ip-10-0-135-61 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:57:52.324800 ip-10-0-135-61 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:52.324800 ip-10-0-135-61 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:57:52.324800 ip-10-0-135-61 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:57:52.326332 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.326257    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:57:52.329059 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329045    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.329059 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329059    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329062    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329066    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329069    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329072    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329075    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329078    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329080    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329083    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329086    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329088    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329091    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329094    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329099    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329102    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329105    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329107    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329110    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329113    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329115    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.329121 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329118    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329120    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329124    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329127    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329130    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329133    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329135    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329138    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329141    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329144    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329146    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329149    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329152    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329154    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329158    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329162    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329164    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329167    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329170    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329172    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.329591 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329175    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329177    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329180    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329188    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329190    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329193    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329196    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329198    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329201    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329204    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329206    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329209    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329211    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329215    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329219    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329222    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329225    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329227    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329230    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329232    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.330084 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329235    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329238    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329240    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329244    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329246    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329249    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329252    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329255    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329257    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329260    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329262    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329265    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329267    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329270    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329272    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329275    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329277    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329280    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329285    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329288    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.330561 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329291    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329295    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329297    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329301    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329304    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329307    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329311    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329314    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329316    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329319    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329322    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329324    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329327    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329329    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329332    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329334    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329337    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.329340    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330030    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.331100 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330037    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330040    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330043    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330045    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330048    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330051    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330053    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330056    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330059    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330062    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330065    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330067    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330070    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330073    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330075    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330078    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330081    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330084    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330086    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330090    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.331563 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330092    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330095    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330098    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330101    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330103    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330106    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330109    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330111    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330114    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330117    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330120    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330123    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330125    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330128    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330131    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330133    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330136    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330138    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330141    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.332075 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330143    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330148    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330152    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330155    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330158    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330161    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330164    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330167    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330170    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330173    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330178    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330182    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330185    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330189    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330192    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330194    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330197    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330200    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330202    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.332532 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330205    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330207    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330210    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330213    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330216    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330219    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330222    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330224    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330227    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330229    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330232    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330235    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330237    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330240    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330242    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330245    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330247    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330250    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330252    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330256    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.333011 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330259    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330262    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330264    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330269    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330272    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330277    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330280    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330284    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330287    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330290    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330292    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330295    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330297    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330300    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330303    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330305    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330308    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330310    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330313    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.333507 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330316    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330647    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330651    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330654    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330657    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330660    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330662    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330665    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330668    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330670    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330673    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330676    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330679    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330681    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330684    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330687    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330692    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330694    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330697    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330700    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.333977 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330703    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330705    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330708    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330710    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330713    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330717    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330720    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330723    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330726    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330728    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330731    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330733    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330736    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330738    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330741    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330743    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330747    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330750    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330753    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330755    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.334467 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330758    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330760    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330763    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330765    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330768    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330770    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330773    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330776    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330780    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330783    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330786    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330788    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330791    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330794    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330796    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330799    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330817    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330820    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330823    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.334986 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330826    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330830    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330834    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330837    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330841    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330844    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330847    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330851    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330853    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330856    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330859    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330870    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330873    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330876    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330879    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330881    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330884    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330887    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330890    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.335445 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330892    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330895    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330899    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330902    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330905    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330909    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330912    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330914    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330917    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330920    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330922    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330925    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330927    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330930    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330933    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330935    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330938    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330940    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330943    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330946    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.335946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330949    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.330952    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331018    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331062    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331069    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331073    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331078    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331081    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331085    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331090    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331093    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331096    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331099    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331103    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331108    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331112    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331115    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331119    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331122    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331125    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331130    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331133    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331136    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331139    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:57:52.336424 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331142    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331146    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331149    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331152    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331155    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331158    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331161    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331163    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331167    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331170    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331175    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331179    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331182    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331185    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331188    2562 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331191    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331195    2562 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331198    2562 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331201    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331204    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331207    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331210    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331214    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331217    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331221    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:57:52.337020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331224    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331227    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331230    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331233    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331250    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331254    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331257    2562 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331261    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331265    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331268    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331271    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331274    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331277    2562 flags.go:64] FLAG: --help="false"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331280    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331283    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331286    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331289    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331294    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331297    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331300    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331303    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331306    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331309    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:57:52.337915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331312    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331315    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331318    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331321    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331324    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331327    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331331    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331334    2562 flags.go:64] FLAG: --lock-file=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331337    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331340    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331343    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331349    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331352    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331355    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331357    2562 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331360    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331363    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331366    2562 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331369    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331374    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331377    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331381    2562 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331384    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331387    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331390    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:57:52.338658 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331393    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331397    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331400    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331403    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331410    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331413    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331416    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331420    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331423    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331428    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331431    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331434    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331437    2562 flags.go:64] FLAG: --port="10250"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331442    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331444    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091abc3b1360674f1"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331447    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331450    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331454    2562 flags.go:64] FLAG: --register-node="true"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331457    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331461    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331464    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331466    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331469    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331473    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331476    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:57:52.339319 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331479    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331482    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331485    2562 flags.go:64] FLAG: --runonce="false"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331487    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331491    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331493    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331496    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331499    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331502    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331506    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331509    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331512    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331515    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331518    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331521    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331524    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331527    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331530    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331535    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331537    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331542    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331565    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331569    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331573    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331576    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:57:52.340001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331582    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331585    2562 flags.go:64] FLAG: --v="2"
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331589    2562 flags.go:64] FLAG: --version="false"
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331593    2562 flags.go:64] FLAG: --vmodule=""
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331597    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.331600    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331711    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331716    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331720    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331723    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331726    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331729    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331732    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331735    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331738    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331741    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331745    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331747    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331750    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331753    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331755    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.340608 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331758    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331761    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331763    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331766    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331769    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331771    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331775    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331778    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331780    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331783    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331785    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331788    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331791    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331794    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331796    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331799    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331814    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331818    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331821    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331823    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.341138 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331826    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331829    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331831    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331834    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331837    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331840    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331842    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331846    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331849    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331851    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331854    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331857    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331860    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331862    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331865    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331867    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331870    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331873    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331876    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331878    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.341630 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331881    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331884    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331886    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331889    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331893    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331898    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331901    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331904    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331907    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331909    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331913    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331917    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331920    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331923    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331926    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331929    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331932    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331935    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.342139 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331938    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331942    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331945    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331947    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331950    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331953    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331956    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331958    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331961    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331964    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331967    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331969    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331972    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331975    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331978    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331980    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331983    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331985    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331988    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331991    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.342646 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331994    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.331997    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.332000    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.332003    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.332005    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.332008    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.332013    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.341120    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.341138    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.341912    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.341920    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341975    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341981    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341984    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341988    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.343196 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341991    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341994    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.341997    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342000    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342003    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342006    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342009    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342013    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342016    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342019    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342021    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342024    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342029    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342032    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342035    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342038    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342046    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342049    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.343583 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342052    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342054    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342057    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342059    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342062    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342065    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342067    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342072    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342075    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342078    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342081    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342084    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342088    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342090    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342093    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342096    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342099    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342101    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342104    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.344056 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342106    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342109    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342111    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342114    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342117    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342119    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342122    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342125    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342127    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342130    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342133    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342136    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342139    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342141    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342144    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342148    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342151    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342154    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342156    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342159    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.344530 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342162    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342164    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342167    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342170    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342172    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342175    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342178    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342180    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342182    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342185    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342188    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342190    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342193    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342195    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342198    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342200    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342203    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342211    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342213    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342216    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.345054 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342219    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342222    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342225    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342228    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342246    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342250    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342254    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342258    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342261    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342264    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342267    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342270    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342273    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342276    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342278    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342281    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.345550 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342284    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.342289    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342406    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342412    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342417    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342421    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342425    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342428    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342431    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342434    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342436    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342439    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342442    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342444    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342448    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342451    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:57:52.346049 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342454    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342456    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342460    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342462    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342465    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342467    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342470    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342473    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342476    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342478    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342481    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342484    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342488    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342492    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342495    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342498    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342500    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342503    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342506    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:57:52.346473 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342509    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342511    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342514    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342517    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342520    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342522    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342525    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342528    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342530    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342533    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342535    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342538    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342542    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342544    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342547    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342550    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342554    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342556    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342559    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342561    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:57:52.347070 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342564    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342567    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342570    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342574    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342577    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342580    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342583    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342585    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342588    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342591    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342593    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342596    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342598    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342601    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342604    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342606    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342608    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342611    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342614    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:57:52.347564 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342617    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342619    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342622    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342624    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342627    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342629    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342632    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342635    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342638    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342640    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342644    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342648    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342651    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342654    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342657    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342660    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342663    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342666    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342669    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:57:52.348338 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342671    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342674    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342676    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342679    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342681    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342684    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342687    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:52.342689    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.342694    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.342787    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.344801    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.345823    2562 server.go:1019] "Starting client certificate rotation"
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.345936    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:57:52.348998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.345987    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:57:52.349370 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.348208    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:57:52.349443 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.349425    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:57:52.357674 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.357658    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:57:52.360559 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.360544    2562 log.go:25] "Validated CRI v1 image API"
Sep 10 20:57:52.361419 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.361403    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:57:52.364307 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.364287    2562 fs.go:135] Filesystem UUIDs: map[12e9fa34-567a-4d5e-96d0-6228f671c807:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aecbf1bd-34a6-4674-abd4-2b7086b62047:/dev/nvme0n1p3]
Sep 10 20:57:52.364378 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.364306    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:57:52.371344 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.371231    2562 manager.go:216] Machine: {Timestamp:2026-09-10 20:57:52.370006618 +0000 UTC m=+0.290627006 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100742 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23b596e41e658c11ca2e16ee8d2712 SystemUUID:ec23b596-e41e-658c-11ca-2e16ee8d2712 BootID:ed76b875-ec7d-4b4d-8d6c-deb600d1534c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:3b:54:fa:75 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:3b:54:fa:75 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:9e:e1:d3:24:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:57:52.371344 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.371332    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:57:52.371467 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.371405    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:57:52.371939 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.371922    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:57:52.374053 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.374030    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:57:52.374174 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.374058    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-61.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:57:52.375238 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.375227    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:57:52.375270 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.375240    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:57:52.375270 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.375248    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:57:52.375974 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.375964    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:57:52.376755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.376746    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:57:52.380998 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.380985    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:57:52.381105 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381096    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:57:52.381136 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381123    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:57:52.381136 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381133    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:57:52.381198 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381148    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:57:52.381225 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381199    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:57:52.381225 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381209    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:57:52.381577 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381557    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:57:52.381577 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.381577    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:57:52.383647 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.383626    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:57:52.385793 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.385779    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:57:52.385885 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.385826    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:57:52.387154 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387142    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387163    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387170    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387176    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387186    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387191    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:57:52.387205 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387204    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:57:52.387395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387209    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:57:52.387395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387215    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:57:52.387395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387220    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:57:52.387395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387234    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:57:52.387395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387239    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:57:52.387963 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387950    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:57:52.387963 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.387963    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:57:52.389781 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.389754    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:57:52.389781 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.389758    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:57:52.389900 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.389849    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-61.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:57:52.392260 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.392248    2562 server.go:1268] "Started kubelet"
Sep 10 20:57:52.392331 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.392307    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:57:52.392435 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.392392    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:57:52.392523 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.392458    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:57:52.393119 ip-10-0-135-61 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:57:52.393861 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.393847    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:57:52.394124 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.394108    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:57:52.396925 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.395914    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113540dec5cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.392209867 +0000 UTC m=+0.312830244,LastTimestamp:2026-09-10 20:57:52.392209867 +0000 UTC m=+0.312830244,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.397126 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.397111    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:57:52.400054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.400036    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:57:52.400539 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.400519    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:57:52.400749 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.400731    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:52.401398 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401380    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:57:52.401456 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401402    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:57:52.401504 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401474    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:57:52.401504 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401384    2562 factory.go:55] Registering systemd factory
Sep 10 20:57:52.401504 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401494    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:57:52.401642 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401584    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:57:52.401642 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401591    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:57:52.401833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401819    2562 factory.go:153] Registering CRI-O factory
Sep 10 20:57:52.401894 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401835    2562 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:57:52.401894 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401884    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:57:52.402000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401903    2562 factory.go:103] Registering Raw factory
Sep 10 20:57:52.402000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.401917    2562 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:57:52.403465 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.403450    2562 manager.go:349] Starting recovery of all containers
Sep 10 20:57:52.411758 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.411728    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:57:52.411896 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.411861    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-61.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:57:52.414949 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.414768    2562 manager.go:354] Recovery completed
Sep 10 20:57:52.419100 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.419089    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.423722 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.423702    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.423785 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.423731    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.423785 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.423742    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.424269 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424251    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:57:52.424269 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424266    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:57:52.424269 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424268    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.424440 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424285    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:57:52.424915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424898    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.424982 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424932    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.424982 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.424945    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.426294 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.426282    2562 policy_none.go:49] "None policy: Start"
Sep 10 20:57:52.426350 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.426299    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:57:52.426350 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.426310    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:57:52.428144 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.428132    2562 policy_none.go:47] "Start"
Sep 10 20:57:52.432011 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.431920    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bf8f8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.423718796 +0000 UTC m=+0.344339173,LastTimestamp:2026-09-10 20:57:52.423718796 +0000 UTC m=+0.344339173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.434189 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.434167    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4dpg"
Sep 10 20:57:52.436116 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.435199    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bfd3f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.423736311 +0000 UTC m=+0.344356688,LastTimestamp:2026-09-10 20:57:52.423736311 +0000 UTC m=+0.344356688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.442074 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.442056    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m4dpg"
Sep 10 20:57:52.444527 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.444451    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bff7a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.42374544 +0000 UTC m=+0.344365817,LastTimestamp:2026-09-10 20:57:52.42374544 +0000 UTC m=+0.344365817,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.452757 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.452689    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-61.ec2.internal.18d4113542bf8f8c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bf8f8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.423718796 +0000 UTC m=+0.344339173,LastTimestamp:2026-09-10 20:57:52.424915206 +0000 UTC m=+0.345535590,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.460357 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.460341    2562 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.460383    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.460396    2562 server.go:87] "Starting device plugin registration server"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.461077    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.461088    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.461184    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.461254    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.461262    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.461975    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.462024    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.461975    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-61.ec2.internal.18d4113542bfd3f7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bfd3f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.423736311 +0000 UTC m=+0.344356688,LastTimestamp:2026-09-10 20:57:52.424939425 +0000 UTC m=+0.345559803,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.471992    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-61.ec2.internal.18d4113542bff7a0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bff7a0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.42374544 +0000 UTC m=+0.344365817,LastTimestamp:2026-09-10 20:57:52.424948793 +0000 UTC m=+0.345569170,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.477694    2562 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2597 into the system container \"/system.slice\": <nil>"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.479832    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d41135451065e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.462570985 +0000 UTC m=+0.383191361,LastTimestamp:2026-09-10 20:57:52.462570985 +0000 UTC m=+0.383191361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.562190    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.562997    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.563022    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.563034    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.568645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.563062    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.572175 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.572107    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-61.ec2.internal.18d4113542bf8f8c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-61.ec2.internal.18d4113542bf8f8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-61.ec2.internal,UID:ip-10-0-135-61.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-61.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-61.ec2.internal,},FirstTimestamp:2026-09-10 20:57:52.423718796 +0000 UTC m=+0.344339173,LastTimestamp:2026-09-10 20:57:52.563010295 +0000 UTC m=+0.483630673,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-61.ec2.internal,}"
Sep 10 20:57:52.581767 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.581712    2562 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-61.ec2.internal\" not found" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.610978 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.610958    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:57:52.612051 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.612024    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:57:52.612139 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.612069    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:57:52.612139 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.612091    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:57:52.612236 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.612169    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:57:52.613798 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.613776    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:52.616258 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.616242    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-61.ec2.internal\" not found" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.713068 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.713045    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"]
Sep 10 20:57:52.713134 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.713109    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.714660 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.714647    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.714709 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.714677    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.714709 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.714702    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.717021 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.717007    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.717120 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.717107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.717159 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.717143    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.718915 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718901    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.718979 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718929    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.718979 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718940    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.718979 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718901    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.719086 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718981    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.719086 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.718991    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.721259 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.721245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.721325 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.721273    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.723901 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.723851    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.723901 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.723881    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.723901 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.723896    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.748501 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.748484    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-61.ec2.internal\" not found" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.752883 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.752867    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-61.ec2.internal\" not found" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.782625 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.782607    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:57:52.783495 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.783474    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:57:52.783570 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.783500    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:57:52.783570 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.783509    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:57:52.783570 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.783527    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.792574 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.792559    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.792620 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.792577    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-61.ec2.internal\": node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:52.803249 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.803230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.803305 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.803253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.803305 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.803270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ed6d91cc61f87a0eb8bab0854fc763b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-61.ec2.internal\" (UID: \"7ed6d91cc61f87a0eb8bab0854fc763b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.807195 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.807180    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:52.903715 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.903715 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.903715 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ed6d91cc61f87a0eb8bab0854fc763b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-61.ec2.internal\" (UID: \"7ed6d91cc61f87a0eb8bab0854fc763b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.903868 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.903868 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ed6d91cc61f87a0eb8bab0854fc763b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-61.ec2.internal\" (UID: \"7ed6d91cc61f87a0eb8bab0854fc763b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.903868 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:52.903776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6a7ee1afb0fb960c468fd7d20a452f34-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal\" (UID: \"6a7ee1afb0fb960c468fd7d20a452f34\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:52.907765 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:52.907750    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.008234 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:53.008208    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.051378 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.051360    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:53.055195 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.055183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:53.108441 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:53.108413    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.208948 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:53.208896    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.309271 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:53.309243    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.348378 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.348356    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:57:53.348737 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.348481    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:57:53.410371 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:53.410348    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-61.ec2.internal\" not found"
Sep 10 20:57:53.483290 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.483242    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:53.501494 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.501474    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:53.519089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.519072    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:57:53.520758 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.520743    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal"
Sep 10 20:57:53.528183 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.528163    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:57:53.572429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.572398    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:52:52 +0000 UTC" deadline="2028-05-03 14:39:55.286543168 +0000 UTC"
Sep 10 20:57:53.572429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.572427    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14417h42m1.714119451s"
Sep 10 20:57:53.700794 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.700771    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:53.701906 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:53.701859    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6d91cc61f87a0eb8bab0854fc763b.slice/crio-97b348a7fdcb18b47f292bf6cd1e78c19539b2a2f82be28a1c723a7f3e8c6380 WatchSource:0}: Error finding container 97b348a7fdcb18b47f292bf6cd1e78c19539b2a2f82be28a1c723a7f3e8c6380: Status 404 returned error can't find the container with id 97b348a7fdcb18b47f292bf6cd1e78c19539b2a2f82be28a1c723a7f3e8c6380
Sep 10 20:57:53.702293 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:53.702269    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7ee1afb0fb960c468fd7d20a452f34.slice/crio-82b495a0c101486b94783df3d1c5f22a315609472537062cf601eca65447dc0f WatchSource:0}: Error finding container 82b495a0c101486b94783df3d1c5f22a315609472537062cf601eca65447dc0f: Status 404 returned error can't find the container with id 82b495a0c101486b94783df3d1c5f22a315609472537062cf601eca65447dc0f
Sep 10 20:57:53.707127 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.707103    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:57:53.855531 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:53.855461    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:54.382974 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.382943    2562 apiserver.go:52] "Watching apiserver"
Sep 10 20:57:54.392301 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.392082    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:57:54.393380 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.393347    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cvk8k","kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk","openshift-image-registry/node-ca-zp29k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal","openshift-multus/multus-additional-cni-plugins-s64mf","openshift-multus/multus-jjdbq","openshift-multus/network-metrics-daemon-9xrp9","kube-system/konnectivity-agent-vc8dg","openshift-cluster-node-tuning-operator/tuned-m6fgz","openshift-dns/node-resolver-hdmm9","openshift-network-diagnostics/network-check-target-zn4gx","openshift-network-operator/iptables-alerter-r6t5q","openshift-ovn-kubernetes/ovnkube-node-zz5x5"]
Sep 10 20:57:54.396218 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.396196    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.399895 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.399393    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:57:54.399895 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.399612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.401623 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.400467    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:57:54.401623 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.401323    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:57:54.401623 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.401392    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.401623 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.401585    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:57:54.405362 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.405272    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hcmdd\""
Sep 10 20:57:54.405794 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.405774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.406621 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.406068    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.407693 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.407628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.408861 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.408640    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409338    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409672    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pzmx8\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409797    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.410126 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.409952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zgj7m\""
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-cnibin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2e81c4a-6431-443e-b0ae-2666493afbd2-hosts-file\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-kubelet\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-daemon-config\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410490    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zfg\" (UniqueName: \"kubernetes.io/projected/a2e81c4a-6431-443e-b0ae-2666493afbd2-kube-api-access-j7zfg\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-netns\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-device-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6tl\" (UniqueName: \"kubernetes.io/projected/5809878d-b929-4111-960f-d8a312a0f2f1-kube-api-access-fh6tl\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-k8s-cni-cncf-io\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-etc-kubernetes\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410752    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-conf-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-multus-certs\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e81c4a-6431-443e-b0ae-2666493afbd2-tmp-dir\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.410874 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.411531 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.410756    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:57:54.411531 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.411043    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.412041 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.411680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-cni-binary-copy\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.412276 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.412254    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rdjvf\""
Sep 10 20:57:54.412965 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.412940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-socket-dir-parent\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.413105 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.413523 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413501    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-bin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.413608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65d6t\" (UniqueName: \"kubernetes.io/projected/ca3b6042-30a3-489f-a16c-ac57565916c1-kube-api-access-65d6t\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.413608 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.413721 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-system-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.413774 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-multus\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.413845 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.413829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-hostroot\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.415044 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.415020    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.415147 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.415111    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l7nkj\""
Sep 10 20:57:54.416299 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.415847    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:57:54.416299 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.416095    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.416666 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.416647    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:57:54.416752 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.416691    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.416989 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.416973    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2tmcl\""
Sep 10 20:57:54.417081 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.417049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:57:54.417221 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.417203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:57:54.417291 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.417245    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:57:54.417554 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.417537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:57:54.418650 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.418179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.419194 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.419179    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g97jb\""
Sep 10 20:57:54.419518 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.419502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:57:54.419778 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.419711    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:57:54.421058 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.421039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.421169 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.421120    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:57:54.422483 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.422462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w75tm\""
Sep 10 20:57:54.422594 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.422482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.422594 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.422487    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.422594 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.422552    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:57:54.423704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.423681    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.426487 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.426469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.426586 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.426534    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:57:54.427560 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.427543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rxq64\""
Sep 10 20:57:54.427655 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.427543    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:57:54.427840 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.427818    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:57:54.428325 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.428303    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:57:54.429257 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.429082    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:54.429257 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.429151    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:57:54.503086 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.503061    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:57:54.514066 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-conf-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514188 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e81c4a-6431-443e-b0ae-2666493afbd2-tmp-dir\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.514188 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-cni-binary-copy\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514188 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-env-overrides\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.514188 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-conf-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514393 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.514393 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-system-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514393 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-system-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514471    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-multus\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514477    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2e81c4a-6431-443e-b0ae-2666493afbd2-tmp-dir\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brp55\" (UniqueName: \"kubernetes.io/projected/d82ca890-8cb0-42e1-9325-555f07de811d-kube-api-access-brp55\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-modprobe-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514575    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-multus\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-cnibin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.514643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-cnibin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-cnibin\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-sys-fs\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/886749a3-399a-4a72-b975-05124965b591-konnectivity-ca\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-etc-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-ovn\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-cni-binary-copy\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-socket-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-host\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-tmp\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.514934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-daemon-config\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.515039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515029    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zfg\" (UniqueName: \"kubernetes.io/projected/a2e81c4a-6431-443e-b0ae-2666493afbd2-kube-api-access-j7zfg\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-registration-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6tl\" (UniqueName: \"kubernetes.io/projected/5809878d-b929-4111-960f-d8a312a0f2f1-kube-api-access-fh6tl\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-kubelet-config\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515527    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-system-cni-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515553    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-serviceca\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-netd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515603    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-config\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-node-log\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-log-socket\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-script-lib\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.515819 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515893    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-var-lib-kubelet\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-multus-certs\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-daemon-config\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-multus-certs\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.515966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516037    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-socket-dir-parent\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-bin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-socket-dir-parent\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65d6t\" (UniqueName: \"kubernetes.io/projected/ca3b6042-30a3-489f-a16c-ac57565916c1-kube-api-access-65d6t\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-cni-bin\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-dbus\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovn-node-metrics-cert\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-hostroot\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516239    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-kubelet\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.516429 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-systemd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-var-lib-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fk7\" (UniqueName: \"kubernetes.io/projected/9ffe6511-d5b0-44e9-9fa8-c75f76855926-kube-api-access-v8fk7\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-lib-modules\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-host\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-hostroot\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-systemd-units\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2e81c4a-6431-443e-b0ae-2666493afbd2-hosts-file\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-kubelet\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhnw\" (UniqueName: \"kubernetes.io/projected/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-kube-api-access-6bhnw\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-var-lib-kubelet\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2e81c4a-6431-443e-b0ae-2666493afbd2-hosts-file\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-bin\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516632    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-netns\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.517354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516737    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-netns\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-kubernetes\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-run\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-sys\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-tuned\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/886749a3-399a-4a72-b975-05124965b591-agent-certs\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-slash\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.516977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-systemd\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-device-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-k8s-cni-cncf-io\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-etc-kubernetes\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.518330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mn2\" (UniqueName: \"kubernetes.io/projected/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-kube-api-access-96mn2\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-conf\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-host-run-k8s-cni-cncf-io\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfp74\" (UniqueName: \"kubernetes.io/projected/405d5d85-57cc-425c-8a7f-9e4b96d68baa-kube-api-access-qfp74\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517202    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5809878d-b929-4111-960f-d8a312a0f2f1-device-dir\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-etc-kubernetes\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517273    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w5rj\" (UniqueName: \"kubernetes.io/projected/4bd50d37-f232-4a31-bd09-b30164a06d08-kube-api-access-9w5rj\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-iptables-alerter-script\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-host-slash\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca3b6042-30a3-489f-a16c-ac57565916c1-multus-cni-dir\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-netns\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.519340 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.517436    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysconfig\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.521109 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.521078    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:57:54.525601 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.525579    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65d6t\" (UniqueName: \"kubernetes.io/projected/ca3b6042-30a3-489f-a16c-ac57565916c1-kube-api-access-65d6t\") pod \"multus-jjdbq\" (UID: \"ca3b6042-30a3-489f-a16c-ac57565916c1\") " pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.525683 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.525574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6tl\" (UniqueName: \"kubernetes.io/projected/5809878d-b929-4111-960f-d8a312a0f2f1-kube-api-access-fh6tl\") pod \"aws-ebs-csi-driver-node-8znzk\" (UID: \"5809878d-b929-4111-960f-d8a312a0f2f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.525683 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.525584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zfg\" (UniqueName: \"kubernetes.io/projected/a2e81c4a-6431-443e-b0ae-2666493afbd2-kube-api-access-j7zfg\") pod \"node-resolver-hdmm9\" (UID: \"a2e81c4a-6431-443e-b0ae-2666493afbd2\") " pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.576687 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.576660    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfvc9"
Sep 10 20:57:54.585659 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.585635    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sfvc9"
Sep 10 20:57:54.616525 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.616484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal" event={"ID":"7ed6d91cc61f87a0eb8bab0854fc763b","Type":"ContainerStarted","Data":"97b348a7fdcb18b47f292bf6cd1e78c19539b2a2f82be28a1c723a7f3e8c6380"}
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-dbus\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovn-node-metrics-cert\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-kubelet\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-systemd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617880    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-var-lib-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fk7\" (UniqueName: \"kubernetes.io/projected/9ffe6511-d5b0-44e9-9fa8-c75f76855926-kube-api-access-v8fk7\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-lib-modules\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-host\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.617980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-systemd-units\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhnw\" (UniqueName: \"kubernetes.io/projected/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-kube-api-access-6bhnw\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618035    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-bin\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.620054 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-kubernetes\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-run\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-sys\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-tuned\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/886749a3-399a-4a72-b975-05124965b591-agent-certs\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618310    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-slash\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-systemd\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618387    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal" event={"ID":"6a7ee1afb0fb960c468fd7d20a452f34","Type":"ContainerStarted","Data":"82b495a0c101486b94783df3d1c5f22a315609472537062cf601eca65447dc0f"}
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mn2\" (UniqueName: \"kubernetes.io/projected/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-kube-api-access-96mn2\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-conf\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfp74\" (UniqueName: \"kubernetes.io/projected/405d5d85-57cc-425c-8a7f-9e4b96d68baa-kube-api-access-qfp74\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-run\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w5rj\" (UniqueName: \"kubernetes.io/projected/4bd50d37-f232-4a31-bd09-b30164a06d08-kube-api-access-9w5rj\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-slash\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618491    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-dbus\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.621007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-iptables-alerter-script\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-host-slash\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-sys\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-netns\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618599    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysconfig\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-lib-modules\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-env-overrides\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brp55\" (UniqueName: \"kubernetes.io/projected/d82ca890-8cb0-42e1-9325-555f07de811d-kube-api-access-brp55\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-host\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-systemd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-modprobe-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-kubelet\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-cnibin\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618763    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-var-lib-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-systemd-units\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.622001 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-cnibin\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/886749a3-399a-4a72-b975-05124965b591-konnectivity-ca\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-etc-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.618870    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-ovn\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-host\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-tmp\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.618993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-kubelet-config\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-netns\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619074    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysconfig\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619105    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-system-cni-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-systemd\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-bin\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-conf\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619540    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-env-overrides\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.622866 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-binary-copy\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-ovn\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-etc-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-host\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-modprobe-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-run-ovn-kubernetes\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-host-slash\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-kubernetes\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bd50d37-f232-4a31-bd09-b30164a06d08-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.620569    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.620644    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:55.120611666 +0000 UTC m=+3.041232050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.619055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bd50d37-f232-4a31-bd09-b30164a06d08-system-cni-dir\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-serviceca\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-netd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-config\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-node-log\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.623755 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-log-socket\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-script-lib\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-var-lib-kubelet\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.621000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-var-lib-kubelet\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.621508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-node-log\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.621694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/886749a3-399a-4a72-b975-05124965b591-konnectivity-ca\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.621998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-config\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.620768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-iptables-alerter-script\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-host-cni-netd\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-run-openvswitch\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9ffe6511-d5b0-44e9-9fa8-c75f76855926-log-socket\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622004    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-serviceca\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca93feb-5081-4d12-ba13-def3a077f69c-kubelet-config\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.622162    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:55.12214398 +0000 UTC m=+3.042764351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622203    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovnkube-script-lib\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.622260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-sysctl-d\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.624565 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.623124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-etc-tuned\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.625464 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.623544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ffe6511-d5b0-44e9-9fa8-c75f76855926-ovn-node-metrics-cert\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.625464 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.624393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/886749a3-399a-4a72-b975-05124965b591-agent-certs\") pod \"konnectivity-agent-vc8dg\" (UID: \"886749a3-399a-4a72-b975-05124965b591\") " pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.626133 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.626096    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/405d5d85-57cc-425c-8a7f-9e4b96d68baa-tmp\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.628729 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.628705    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:54.628868 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.628752    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:54.628868 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.628771    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:54.628868 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:54.628861    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:57:55.128841568 +0000 UTC m=+3.049461957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:54.631607 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.631472    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w5rj\" (UniqueName: \"kubernetes.io/projected/4bd50d37-f232-4a31-bd09-b30164a06d08-kube-api-access-9w5rj\") pod \"multus-additional-cni-plugins-s64mf\" (UID: \"4bd50d37-f232-4a31-bd09-b30164a06d08\") " pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.631728 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.631641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhnw\" (UniqueName: \"kubernetes.io/projected/f0c89aed-6aa8-44c7-acd8-d747faa06ff8-kube-api-access-6bhnw\") pod \"iptables-alerter-r6t5q\" (UID: \"f0c89aed-6aa8-44c7-acd8-d747faa06ff8\") " pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.633036 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.632986    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfp74\" (UniqueName: \"kubernetes.io/projected/405d5d85-57cc-425c-8a7f-9e4b96d68baa-kube-api-access-qfp74\") pod \"tuned-m6fgz\" (UID: \"405d5d85-57cc-425c-8a7f-9e4b96d68baa\") " pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:54.633134 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.633030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fk7\" (UniqueName: \"kubernetes.io/projected/9ffe6511-d5b0-44e9-9fa8-c75f76855926-kube-api-access-v8fk7\") pod \"ovnkube-node-zz5x5\" (UID: \"9ffe6511-d5b0-44e9-9fa8-c75f76855926\") " pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.633335 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.633311    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mn2\" (UniqueName: \"kubernetes.io/projected/2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f-kube-api-access-96mn2\") pod \"node-ca-zp29k\" (UID: \"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f\") " pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.636378 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.636352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brp55\" (UniqueName: \"kubernetes.io/projected/d82ca890-8cb0-42e1-9325-555f07de811d-kube-api-access-brp55\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:54.719013 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.718981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjdbq"
Sep 10 20:57:54.727239 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.727216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk"
Sep 10 20:57:54.739137 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.739117    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:57:54.744974 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.744953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hdmm9"
Sep 10 20:57:54.752630 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.752608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zp29k"
Sep 10 20:57:54.761546 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.761525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:57:54.769481 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.769460    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s64mf"
Sep 10 20:57:54.778249 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.778216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6t5q"
Sep 10 20:57:54.784000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:54.783981    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m6fgz"
Sep 10 20:57:55.125526 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:55.125422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:55.125526 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:55.125493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:55.125746 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.125595    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:55.125746 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.125616    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:55.125746 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.125678    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:56.125656349 +0000 UTC m=+4.046276729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:55.125746 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.125698    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:56.125687994 +0000 UTC m=+4.046308362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:55.226401 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:55.226370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:55.226574 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.226517    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:55.226574 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.226540    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:55.226574 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.226551    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:55.226718 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:55.226599    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:57:56.226586061 +0000 UTC m=+4.147206440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:55.590572 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:55.590278    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:52:55 +0000 UTC" deadline="2028-02-11 13:44:30.773756777 +0000 UTC"
Sep 10 20:57:55.590572 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:55.590528    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12448h46m35.183236815s"
Sep 10 20:57:55.674194 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.674065    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5809878d_b929_4111_960f_d8a312a0f2f1.slice/crio-ed118e040003d4199104e8e436c2d6122fd65083920a663648900c74a6949b14 WatchSource:0}: Error finding container ed118e040003d4199104e8e436c2d6122fd65083920a663648900c74a6949b14: Status 404 returned error can't find the container with id ed118e040003d4199104e8e436c2d6122fd65083920a663648900c74a6949b14
Sep 10 20:57:55.684384 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.684356    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe6511_d5b0_44e9_9fa8_c75f76855926.slice/crio-dd64a37ab5f7ebea8aefcc057a87e981f9b28b85202451c2f4b143c31c092861 WatchSource:0}: Error finding container dd64a37ab5f7ebea8aefcc057a87e981f9b28b85202451c2f4b143c31c092861: Status 404 returned error can't find the container with id dd64a37ab5f7ebea8aefcc057a87e981f9b28b85202451c2f4b143c31c092861
Sep 10 20:57:55.685515 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.685426    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df000b5_6b2e_42f9_b1ab_8ac8ff03da1f.slice/crio-40e0a3d47c6ed814b03bd53bf3dc7552d43a87d2c96c9656cf6b79a8ef5047ce WatchSource:0}: Error finding container 40e0a3d47c6ed814b03bd53bf3dc7552d43a87d2c96c9656cf6b79a8ef5047ce: Status 404 returned error can't find the container with id 40e0a3d47c6ed814b03bd53bf3dc7552d43a87d2c96c9656cf6b79a8ef5047ce
Sep 10 20:57:55.689050 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.689028    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d5d85_57cc_425c_8a7f_9e4b96d68baa.slice/crio-83ff669976fcaaa0522f607b5adc0edd3282263a5e1938a6b55c1377b1e93a30 WatchSource:0}: Error finding container 83ff669976fcaaa0522f607b5adc0edd3282263a5e1938a6b55c1377b1e93a30: Status 404 returned error can't find the container with id 83ff669976fcaaa0522f607b5adc0edd3282263a5e1938a6b55c1377b1e93a30
Sep 10 20:57:55.690308 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.690212    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3b6042_30a3_489f_a16c_ac57565916c1.slice/crio-daeec07ba0e3c5ce25245bcdce235830bd599b33edc8948ccb6af2dc1cd6dd07 WatchSource:0}: Error finding container daeec07ba0e3c5ce25245bcdce235830bd599b33edc8948ccb6af2dc1cd6dd07: Status 404 returned error can't find the container with id daeec07ba0e3c5ce25245bcdce235830bd599b33edc8948ccb6af2dc1cd6dd07
Sep 10 20:57:55.691754 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.691701    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886749a3_399a_4a72_b975_05124965b591.slice/crio-78db5a730c82b106140242f7d56602c4a785d559d7bb98486d81baa4f1543a40 WatchSource:0}: Error finding container 78db5a730c82b106140242f7d56602c4a785d559d7bb98486d81baa4f1543a40: Status 404 returned error can't find the container with id 78db5a730c82b106140242f7d56602c4a785d559d7bb98486d81baa4f1543a40
Sep 10 20:57:55.692946 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.692923    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e81c4a_6431_443e_b0ae_2666493afbd2.slice/crio-b0745c0611abe7014028cd3506e3334745195ddf386eb149555921c0444e5ba5 WatchSource:0}: Error finding container b0745c0611abe7014028cd3506e3334745195ddf386eb149555921c0444e5ba5: Status 404 returned error can't find the container with id b0745c0611abe7014028cd3506e3334745195ddf386eb149555921c0444e5ba5
Sep 10 20:57:55.693667 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.693639    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd50d37_f232_4a31_bd09_b30164a06d08.slice/crio-da763e0e3d16d7b397c246bbf885a4fb9a341313498ad60c931c395e191c0439 WatchSource:0}: Error finding container da763e0e3d16d7b397c246bbf885a4fb9a341313498ad60c931c395e191c0439: Status 404 returned error can't find the container with id da763e0e3d16d7b397c246bbf885a4fb9a341313498ad60c931c395e191c0439
Sep 10 20:57:55.694906 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:57:55.694868    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c89aed_6aa8_44c7_acd8_d747faa06ff8.slice/crio-fe1959253445636fe548dada3ca19c4aba27870d51b97c0580fa292a7ef4ebbc WatchSource:0}: Error finding container fe1959253445636fe548dada3ca19c4aba27870d51b97c0580fa292a7ef4ebbc: Status 404 returned error can't find the container with id fe1959253445636fe548dada3ca19c4aba27870d51b97c0580fa292a7ef4ebbc
Sep 10 20:57:56.134458 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.134419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:56.134641 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.134477    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:56.134641 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.134627    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:56.134766 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.134695    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:57:58.134674659 +0000 UTC m=+6.055295031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:56.135144 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.135122    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:56.135322 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.135186    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:57:58.135169052 +0000 UTC m=+6.055789432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:56.235775 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.235697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:56.235941 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.235873    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:56.235941 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.235895    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:56.235941 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.235908    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:56.236111 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.235971    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:57:58.235952101 +0000 UTC m=+6.156572468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:56.590845 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.590741    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:52:55 +0000 UTC" deadline="2028-05-08 20:35:19.029167366 +0000 UTC"
Sep 10 20:57:56.590845 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.590782    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14543h37m22.438390606s"
Sep 10 20:57:56.614020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.613996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:56.614161 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.614127    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:57:56.614562 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.614540    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:56.614673 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.614641    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:57:56.614831 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.614799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:56.614940 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:56.614920    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:57:56.627953 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.627925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6t5q" event={"ID":"f0c89aed-6aa8-44c7-acd8-d747faa06ff8","Type":"ContainerStarted","Data":"fe1959253445636fe548dada3ca19c4aba27870d51b97c0580fa292a7ef4ebbc"}
Sep 10 20:57:56.632873 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.632843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vc8dg" event={"ID":"886749a3-399a-4a72-b975-05124965b591","Type":"ContainerStarted","Data":"78db5a730c82b106140242f7d56602c4a785d559d7bb98486d81baa4f1543a40"}
Sep 10 20:57:56.638195 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.638170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjdbq" event={"ID":"ca3b6042-30a3-489f-a16c-ac57565916c1","Type":"ContainerStarted","Data":"daeec07ba0e3c5ce25245bcdce235830bd599b33edc8948ccb6af2dc1cd6dd07"}
Sep 10 20:57:56.643803 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.643777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m6fgz" event={"ID":"405d5d85-57cc-425c-8a7f-9e4b96d68baa","Type":"ContainerStarted","Data":"83ff669976fcaaa0522f607b5adc0edd3282263a5e1938a6b55c1377b1e93a30"}
Sep 10 20:57:56.657376 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.657350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"dd64a37ab5f7ebea8aefcc057a87e981f9b28b85202451c2f4b143c31c092861"}
Sep 10 20:57:56.664480 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.664453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zp29k" event={"ID":"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f","Type":"ContainerStarted","Data":"40e0a3d47c6ed814b03bd53bf3dc7552d43a87d2c96c9656cf6b79a8ef5047ce"}
Sep 10 20:57:56.668757 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.668678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk" event={"ID":"5809878d-b929-4111-960f-d8a312a0f2f1","Type":"ContainerStarted","Data":"ed118e040003d4199104e8e436c2d6122fd65083920a663648900c74a6949b14"}
Sep 10 20:57:56.674752 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.674723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal" event={"ID":"7ed6d91cc61f87a0eb8bab0854fc763b","Type":"ContainerStarted","Data":"cd53924c2d25fcb3890d9a4637fbab07d8819b8d95ef003cbba5490315a9aede"}
Sep 10 20:57:56.689374 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.689262    2562 generic.go:356] "Generic (PLEG): container finished" podID="6a7ee1afb0fb960c468fd7d20a452f34" containerID="ce72017778510ee85282178f73b933e3b4e17b28af95dfff5f7bffbd112009f1" exitCode=0
Sep 10 20:57:56.689374 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.689346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal" event={"ID":"6a7ee1afb0fb960c468fd7d20a452f34","Type":"ContainerDied","Data":"ce72017778510ee85282178f73b933e3b4e17b28af95dfff5f7bffbd112009f1"}
Sep 10 20:57:56.691501 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.691193    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-61.ec2.internal" podStartSLOduration=2.691175733 podStartE2EDuration="2.691175733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:56.690113316 +0000 UTC m=+4.610733711" watchObservedRunningTime="2026-09-10 20:57:56.691175733 +0000 UTC m=+4.611796126"
Sep 10 20:57:56.723315 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.723285    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerStarted","Data":"da763e0e3d16d7b397c246bbf885a4fb9a341313498ad60c931c395e191c0439"}
Sep 10 20:57:56.741269 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:56.740978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hdmm9" event={"ID":"a2e81c4a-6431-443e-b0ae-2666493afbd2","Type":"ContainerStarted","Data":"b0745c0611abe7014028cd3506e3334745195ddf386eb149555921c0444e5ba5"}
Sep 10 20:57:57.776483 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:57.776400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal" event={"ID":"6a7ee1afb0fb960c468fd7d20a452f34","Type":"ContainerStarted","Data":"b0f7073d2013dfa43ebd144ac4fc17c45bda046d9a5c62aab4667e05e79675ce"}
Sep 10 20:57:58.153127 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.153085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:58.153326 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.153177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:58.153418 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.153395    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:58.153488 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.153475    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:02.153454026 +0000 UTC m=+10.074074412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:57:58.153946 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.153924    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:58.154047 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.153980    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:58:02.153964781 +0000 UTC m=+10.074585147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:57:58.253923 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.253852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:58.254100 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.254049    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:57:58.254100 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.254073    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:57:58.254100 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.254089    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:58.254281 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.254155    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:02.25413526 +0000 UTC m=+10.174755631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.613061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.613061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.613218    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.613292    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:57:58.613357    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:57:58.613625 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:57:58.613433    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:00.615908 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:00.615435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:00.615908 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:00.615564    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:00.615908 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:00.615907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:00.616489 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:00.616038    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:00.616489 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:00.616049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:00.616489 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:00.616141    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:02.191972 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.191574    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:02.191972 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.191630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:02.191972 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.191766    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:02.191972 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.191840    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:58:10.191820615 +0000 UTC m=+18.112440996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:02.192601 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.191990    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:02.192601 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.192054    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:10.19203776 +0000 UTC m=+18.112658126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:02.292736 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.292170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:02.292736 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.292324    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:02.292736 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.292346    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:02.292736 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.292359    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:02.292736 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.292415    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:10.292397456 +0000 UTC m=+18.213017844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:02.614020 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.613936    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:02.614175 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.614049    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:02.614468 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.614450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:02.614557 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.614538    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:02.614825 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:02.614621    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:02.614825 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:02.614733    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:04.612638 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:04.612352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:04.613077 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:04.612765    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:04.613077 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:04.612449    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:04.613077 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:04.612352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:04.613077 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:04.612932    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:04.613077 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:04.613061    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:06.612997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:06.612916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:06.613463 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:06.612916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:06.613463 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:06.613042    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:06.613463 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:06.613107    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:06.613463 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:06.612923    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:06.613463 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:06.613192    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:08.613167 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:08.613077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:08.613611 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:08.613083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:08.613611 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:08.613205    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:08.613611 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:08.613232    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:08.613611 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:08.613316    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:08.613611 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:08.613422    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:10.248006 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.247966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:10.248512 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.248069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:10.248512 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.248144    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:10.248512 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.248168    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:10.248512 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.248220    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:58:26.248198895 +0000 UTC m=+34.168819261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:10.248512 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.248241    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:26.248232369 +0000 UTC m=+34.168852738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:10.349042 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.349010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:10.349198 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.349173    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:10.349198 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.349196    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:10.349320 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.349205    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:10.349320 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.349272    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:26.349236441 +0000 UTC m=+34.269856805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:10.613166 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.613094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:10.613328 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.613094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:10.613328 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.613224    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:10.613328 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:10.613094    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:10.613328 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.613295    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:10.613545 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:10.613347    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:12.614080 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:12.614046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:12.614533 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:12.614149    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:12.614533 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:12.614157    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:12.614533 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:12.614178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:12.614533 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:12.614287    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:12.614533 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:12.614437    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:13.808641 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.808451    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vc8dg" event={"ID":"886749a3-399a-4a72-b975-05124965b591","Type":"ContainerStarted","Data":"05b85add33f2404b5174d92d9b27fbd5148356579c4c9492cd43af06fdbe8407"}
Sep 10 20:58:13.810013 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.809990    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjdbq" event={"ID":"ca3b6042-30a3-489f-a16c-ac57565916c1","Type":"ContainerStarted","Data":"761574b2d09ff6c86700a9b644ab7dd3c263e51f00c31f2aac707636aa9e637a"}
Sep 10 20:58:13.811473 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.811450    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m6fgz" event={"ID":"405d5d85-57cc-425c-8a7f-9e4b96d68baa","Type":"ContainerStarted","Data":"dd7d5c5d55c288a9fc730394ef1b8d3ebaf31eccd38584403075923aaf640793"}
Sep 10 20:58:13.813404 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.813384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"b910b02b06cdb6053e77eb80bf646de1aecd03f2b2fd8c00ebb77f434727b994"}
Sep 10 20:58:13.813478 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.813442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"be946545e19be5b0cdbc3e6bad4e4a928ea5ec77ec4129b056e5b252aae0caa0"}
Sep 10 20:58:13.814905 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.814884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zp29k" event={"ID":"2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f","Type":"ContainerStarted","Data":"082fed099d14ebc18ba6c0bf74511400a3d4088704c1fdf42296aa481a88a1a8"}
Sep 10 20:58:13.816377 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.816353    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk" event={"ID":"5809878d-b929-4111-960f-d8a312a0f2f1","Type":"ContainerStarted","Data":"e573bc6b18828b72e66272fdbea012c68d4c6d27465128fa24ec67338c553781"}
Sep 10 20:58:13.817800 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.817778    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="51170266b2e70b578f096b5fb32c7f7c01f74d15ee480c17c8e1de33041078df" exitCode=0
Sep 10 20:58:13.817891 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.817853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"51170266b2e70b578f096b5fb32c7f7c01f74d15ee480c17c8e1de33041078df"}
Sep 10 20:58:13.819260 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.819242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hdmm9" event={"ID":"a2e81c4a-6431-443e-b0ae-2666493afbd2","Type":"ContainerStarted","Data":"4b0223f28737017241f5f6afba2bf6f77467ad15977b6e7ac732d58771f31ad6"}
Sep 10 20:58:13.823037 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.822998    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-61.ec2.internal" podStartSLOduration=19.822986754 podStartE2EDuration="19.822986754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:57:57.793331294 +0000 UTC m=+5.713951687" watchObservedRunningTime="2026-09-10 20:58:13.822986754 +0000 UTC m=+21.743607139"
Sep 10 20:58:13.823235 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.823215    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vc8dg" podStartSLOduration=3.25066994 podStartE2EDuration="20.823208007s" totalImagesPullingTime="17.572538067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:13.822887309 +0000 UTC m=+21.743507728" watchObservedRunningTime="2026-09-10 20:58:13.823208007 +0000 UTC m=+21.743828393"
Sep 10 20:58:13.838780 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.838740    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hdmm9" podStartSLOduration=3.059893232 podStartE2EDuration="20.838728746s" totalImagesPullingTime="17.778835514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:13.838321464 +0000 UTC m=+21.758941863" watchObservedRunningTime="2026-09-10 20:58:13.838728746 +0000 UTC m=+21.759349131"
Sep 10 20:58:13.852406 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.852353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zp29k" podStartSLOduration=3.273786371 podStartE2EDuration="20.852333592s" totalImagesPullingTime="17.578547221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:13.851499456 +0000 UTC m=+21.772119863" watchObservedRunningTime="2026-09-10 20:58:13.852333592 +0000 UTC m=+21.772953977"
Sep 10 20:58:13.885896 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.885856    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m6fgz" podStartSLOduration=3.078559506 podStartE2EDuration="20.885840609s" totalImagesPullingTime="17.807281103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:13.885704591 +0000 UTC m=+21.806324977" watchObservedRunningTime="2026-09-10 20:58:13.885840609 +0000 UTC m=+21.806460988"
Sep 10 20:58:13.902395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:13.902356    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jjdbq" podStartSLOduration=3.01751999 podStartE2EDuration="20.902341072s" totalImagesPullingTime="17.884821082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:13.90229356 +0000 UTC m=+21.822913966" watchObservedRunningTime="2026-09-10 20:58:13.902341072 +0000 UTC m=+21.822961458"
Sep 10 20:58:14.612833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.612790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:14.613012 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.612790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:14.613012 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:14.612947    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:14.613012 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.612972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:14.613186 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:14.613087    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:14.613248 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:14.613205    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:14.824222 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.824146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"61e7d6451989776786e5b47f1ca7b4ee0fd0842b4bada715393cd659259fc968"}
Sep 10 20:58:14.824833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.824262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"6077fb8eb94ab0633f3e7d23e7b1e24eebbb296c9ac8aabfa7dc1743345fc5f7"}
Sep 10 20:58:14.824833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.824313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"eff89dd998310cc5b7963ce6640553fc82273752c30314708e971d0a399aa7b5"}
Sep 10 20:58:14.824833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:14.824360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"dc289f17d90e2896bddde5f3f15f348b63b153f537a97261784ff090c9c1aac6"}
Sep 10 20:58:15.173127 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.172874    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:58:15.470579 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.470375    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:58:15.173120462Z","UUID":"a7c412f5-5de3-423d-b22d-7b02da167040","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:58:15.472090 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.472069    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:58:15.472204 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.472102    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:58:15.827883 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.827791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk" event={"ID":"5809878d-b929-4111-960f-d8a312a0f2f1","Type":"ContainerStarted","Data":"3ab09bd468a6c3894f12ee316f738b754147a7007c5547856a6ce320f9b29303"}
Sep 10 20:58:15.829584 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.829557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6t5q" event={"ID":"f0c89aed-6aa8-44c7-acd8-d747faa06ff8","Type":"ContainerStarted","Data":"667782561c7915bb8e7165867003d99b7b9371679379adbda9b55582fdf04fde"}
Sep 10 20:58:15.844781 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:15.844731    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6t5q" podStartSLOduration=5.10571696 podStartE2EDuration="22.844711738s" totalImagesPullingTime="17.738994778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:15.843932733 +0000 UTC m=+23.764553118" watchObservedRunningTime="2026-09-10 20:58:15.844711738 +0000 UTC m=+23.765332125"
Sep 10 20:58:16.613372 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.613271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:16.613372 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.613298    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:16.613372 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.613328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:16.613632 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:16.613435    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:16.613632 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:16.613578    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:16.613743 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:16.613678    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:16.836833 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.836786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"0f80f1a61783f418264c1d6103a8c4c5bad94559beed37f07fadf4d8f3b5063f"}
Sep 10 20:58:16.838829 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.838789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk" event={"ID":"5809878d-b929-4111-960f-d8a312a0f2f1","Type":"ContainerStarted","Data":"51193a06b3ca4ee9f3bcdaebe296a0844b20a15d362c596ce1cce9af52dc9e16"}
Sep 10 20:58:16.856599 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:16.856550    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8znzk" podStartSLOduration=3.736769845 podStartE2EDuration="23.856534661s" totalImagesPullingTime="20.119764816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:16.856487058 +0000 UTC m=+24.777107469" watchObservedRunningTime="2026-09-10 20:58:16.856534661 +0000 UTC m=+24.777155071"
Sep 10 20:58:18.223187 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.223153    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:58:18.223800 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.223780    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:58:18.613152 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.613126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:18.613152 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.613141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:18.613317 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.613137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:18.613317 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:18.613257    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:18.613398 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:18.613375    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:18.613490 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:18.613470    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:18.851691 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.851497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" event={"ID":"9ffe6511-d5b0-44e9-9fa8-c75f76855926","Type":"ContainerStarted","Data":"0ef5c2fbef073ea4357469ad5454ff10065bbed06f690afede5724a56efd15be"}
Sep 10 20:58:18.851817 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.851754    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:18.851880 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.851822    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:18.851880 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.851855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:18.853876 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.853851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerStarted","Data":"b6d351538975f5300038bb9b2241db8d6b38c27d69d3683766a3fc468f0c7cb7"}
Sep 10 20:58:18.854033 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.854014    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:58:18.854534 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.854514    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vc8dg"
Sep 10 20:58:18.867112 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.867089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:18.867268 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.867253    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:18.878537 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:18.878498    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5" podStartSLOduration=8.005977136 podStartE2EDuration="25.8784841s" totalImagesPullingTime="17.872506964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:18.87802269 +0000 UTC m=+26.798643109" watchObservedRunningTime="2026-09-10 20:58:18.8784841 +0000 UTC m=+26.799104486"
Sep 10 20:58:19.858713 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:19.858679    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="b6d351538975f5300038bb9b2241db8d6b38c27d69d3683766a3fc468f0c7cb7" exitCode=0
Sep 10 20:58:19.859346 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:19.858759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"b6d351538975f5300038bb9b2241db8d6b38c27d69d3683766a3fc468f0c7cb7"}
Sep 10 20:58:20.510050 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.510013    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn4gx"]
Sep 10 20:58:20.510263 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.510176    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:20.510320 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:20.510274    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:20.512621 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.512595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvk8k"]
Sep 10 20:58:20.512742 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.512698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:20.512825 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:20.512791    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:20.513284 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.513264    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xrp9"]
Sep 10 20:58:20.513373 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:20.513351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:20.513441 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:20.513425    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:21.612875 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:21.612629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:21.613398 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:21.612888    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:21.865440 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:21.865359    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="16e1951036335459fc6bae1d71b5614c7e42e82dae6e2b05fb351cb5c037c1c4" exitCode=0
Sep 10 20:58:21.865440 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:21.865426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"16e1951036335459fc6bae1d71b5614c7e42e82dae6e2b05fb351cb5c037c1c4"}
Sep 10 20:58:22.613400 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:22.613372    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:22.613739 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:22.613503    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:22.613739 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:22.613556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:22.613739 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:22.613653    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:23.612585 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:23.612559    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:23.612702 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:23.612677    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:23.872119 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:23.872047    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="471b6063122567f149d1d463c1fe2d4e5cad38c2a2620c8a57d9c0c42536a3a6" exitCode=0
Sep 10 20:58:23.872119 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:23.872085    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"471b6063122567f149d1d463c1fe2d4e5cad38c2a2620c8a57d9c0c42536a3a6"}
Sep 10 20:58:24.613557 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:24.613012    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:24.613557 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:24.613153    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9xrp9" podUID="d82ca890-8cb0-42e1-9325-555f07de811d"
Sep 10 20:58:24.613557 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:24.613206    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:24.613557 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:24.613292    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zn4gx" podUID="e6bca96e-8693-433d-bec4-e7345926321f"
Sep 10 20:58:25.612942 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:25.612872    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:25.613404 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:25.612970    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvk8k" podUID="eca93feb-5081-4d12-ba13-def3a077f69c"
Sep 10 20:58:26.268439 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.268410    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:26.268629 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.268497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:26.268629 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.268559    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:26.268629 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.268570    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:26.268629 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.268622    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret podName:eca93feb-5081-4d12-ba13-def3a077f69c nodeName:}" failed. No retries permitted until 2026-09-10 20:58:58.26860782 +0000 UTC m=+66.189228201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret") pod "global-pull-secret-syncer-cvk8k" (UID: "eca93feb-5081-4d12-ba13-def3a077f69c") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:26.268781 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.268641    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs podName:d82ca890-8cb0-42e1-9325-555f07de811d nodeName:}" failed. No retries permitted until 2026-09-10 20:58:58.268632938 +0000 UTC m=+66.189253302 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs") pod "network-metrics-daemon-9xrp9" (UID: "d82ca890-8cb0-42e1-9325-555f07de811d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:26.358837 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.358749    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-61.ec2.internal" event="NodeReady"
Sep 10 20:58:26.358981 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.358918    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:58:26.369442 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.369409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:26.369591 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.369562    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:26.369591 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.369581    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:26.369694 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.369595    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dbh28 for pod openshift-network-diagnostics/network-check-target-zn4gx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:26.369694 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.369652    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28 podName:e6bca96e-8693-433d-bec4-e7345926321f nodeName:}" failed. No retries permitted until 2026-09-10 20:58:58.369636678 +0000 UTC m=+66.290257043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dbh28" (UniqueName: "kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28") pod "network-check-target-zn4gx" (UID: "e6bca96e-8693-433d-bec4-e7345926321f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:26.392696 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.392667    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:58:26.395083 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.395057    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.397835 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.397663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:58:26.398138 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.398086    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:58:26.398259 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.398106    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:58:26.398259 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.398170    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-88l8n\""
Sep 10 20:58:26.404633 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.404614    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:58:26.404890 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.404872    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q996r"]
Sep 10 20:58:26.406636 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.406615    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.408714 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.408671    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8sx7m"]
Sep 10 20:58:26.409139 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.409119    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:58:26.409334 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.409312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:58:26.409406 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.409349    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-56bmz\""
Sep 10 20:58:26.410664 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.410646    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:58:26.410742 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.410716    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.413315 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.413293    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:58:26.413407 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.413342    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:58:26.413780 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.413515    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:58:26.413780 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.413546    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jcxqf\""
Sep 10 20:58:26.417067 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.417049    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q996r"]
Sep 10 20:58:26.424035 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.423986    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8sx7m"]
Sep 10 20:58:26.470042 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470235 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470235 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470235 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470381 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470381 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470381 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqqw\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.470503 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.470394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571449 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-config-volume\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.571449 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571689 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571689 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571689 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571689 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqqw\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571689 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7fw\" (UniqueName: \"kubernetes.io/projected/f18c590a-d095-44c2-b581-69fb5f7f00cf-kube-api-access-vq7fw\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f18c590a-d095-44c2-b581-69fb5f7f00cf-cert\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571867    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-tmp-dir\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.571971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.571951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxdq\" (UniqueName: \"kubernetes.io/projected/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-kube-api-access-bxxdq\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.572335 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.572058    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:26.572335 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.572074    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:26.572335 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.572144    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:27.072123517 +0000 UTC m=+34.992743895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:26.572482 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.572461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.572583 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.572558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.576167 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.576142    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.576167 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.576162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.580979 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.580955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.581486 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.581464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqqw\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:26.612641 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.612586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:26.612837 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.612821    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:26.615838 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.615796    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pbskf\""
Sep 10 20:58:26.616157 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.616075    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:26.616358 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.616342    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:26.616558 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.616542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:26.616784 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.616769    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g6p7f\""
Sep 10 20:58:26.672584 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672556    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxdq\" (UniqueName: \"kubernetes.io/projected/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-kube-api-access-bxxdq\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.672727 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-config-volume\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.672790 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672725    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7fw\" (UniqueName: \"kubernetes.io/projected/f18c590a-d095-44c2-b581-69fb5f7f00cf-kube-api-access-vq7fw\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.672859 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f18c590a-d095-44c2-b581-69fb5f7f00cf-cert\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.672859 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.672976 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.672895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-tmp-dir\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.673038 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.672993    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:26.673093 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:26.673068    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:27.173046904 +0000 UTC m=+35.093667268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:26.673174 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.673153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-tmp-dir\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.673229 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.673189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-config-volume\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.675372 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.675345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f18c590a-d095-44c2-b581-69fb5f7f00cf-cert\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.681317 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.681294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxdq\" (UniqueName: \"kubernetes.io/projected/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-kube-api-access-bxxdq\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:26.682434 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.682411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7fw\" (UniqueName: \"kubernetes.io/projected/f18c590a-d095-44c2-b581-69fb5f7f00cf-kube-api-access-vq7fw\") pod \"ingress-canary-8sx7m\" (UID: \"f18c590a-d095-44c2-b581-69fb5f7f00cf\") " pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:26.725540 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:26.725516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8sx7m"
Sep 10 20:58:27.076147 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:27.075859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:27.076312 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:27.076043    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:27.076312 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:27.076221    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:27.076312 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:27.076294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:28.076274403 +0000 UTC m=+35.996894781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:27.177494 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:27.177459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:27.177661 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:27.177606    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:27.177738 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:27.177672    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:28.177652281 +0000 UTC m=+36.098272649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:27.612448 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:27.612413    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:27.616530 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:27.616503    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:27.884988 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:27.884962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerStarted","Data":"24d5d10b6c4a1792af3036b05be29eec82019be7c50c8fc899882a59f8e2fa3b"}
Sep 10 20:58:28.001141 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.001109    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8sx7m"]
Sep 10 20:58:28.005991 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:28.005965    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18c590a_d095_44c2_b581_69fb5f7f00cf.slice/crio-6d80ae7e914a5dc5be5feb8d10e4b56cfeedb63967353fecc37a4bc414cfeca7 WatchSource:0}: Error finding container 6d80ae7e914a5dc5be5feb8d10e4b56cfeedb63967353fecc37a4bc414cfeca7: Status 404 returned error can't find the container with id 6d80ae7e914a5dc5be5feb8d10e4b56cfeedb63967353fecc37a4bc414cfeca7
Sep 10 20:58:28.084749 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.084722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:28.084859 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:28.084849    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:28.084900 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:28.084862    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:28.084938 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:28.084911    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:30.084898461 +0000 UTC m=+38.005518825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:28.185261 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.185190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:28.185386 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:28.185335    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:28.185422 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:28.185403    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:30.185385503 +0000 UTC m=+38.106005888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:28.890568 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.890533    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="24d5d10b6c4a1792af3036b05be29eec82019be7c50c8fc899882a59f8e2fa3b" exitCode=0
Sep 10 20:58:28.891305 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.890613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"24d5d10b6c4a1792af3036b05be29eec82019be7c50c8fc899882a59f8e2fa3b"}
Sep 10 20:58:28.892470 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:28.892442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8sx7m" event={"ID":"f18c590a-d095-44c2-b581-69fb5f7f00cf","Type":"ContainerStarted","Data":"6d80ae7e914a5dc5be5feb8d10e4b56cfeedb63967353fecc37a4bc414cfeca7"}
Sep 10 20:58:29.899029 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:29.898998    2562 generic.go:356] "Generic (PLEG): container finished" podID="4bd50d37-f232-4a31-bd09-b30164a06d08" containerID="35170b7ad957a220e834e2b3b4629dca5cf4c58cb5df2de13a15cc14646c8238" exitCode=0
Sep 10 20:58:29.899531 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:29.899065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerDied","Data":"35170b7ad957a220e834e2b3b4629dca5cf4c58cb5df2de13a15cc14646c8238"}
Sep 10 20:58:30.100479 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.100445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:30.100628 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:30.100551    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:30.100628 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:30.100563    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:30.100628 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:30.100615    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:34.100601049 +0000 UTC m=+42.021221414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:30.201729 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.201525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:30.201854 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:30.201695    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:30.201909 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:30.201892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:34.201869774 +0000 UTC m=+42.122490153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:30.904609 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.904533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s64mf" event={"ID":"4bd50d37-f232-4a31-bd09-b30164a06d08","Type":"ContainerStarted","Data":"d2f4ce3cfda893de7434054c7bc546376ff5db9373b885a20e6fd0361c667bd0"}
Sep 10 20:58:30.906012 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.905989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8sx7m" event={"ID":"f18c590a-d095-44c2-b581-69fb5f7f00cf","Type":"ContainerStarted","Data":"401902819bf6fafec244cf199350a9bb171cfb6e4140989f7cea4f80be059fe6"}
Sep 10 20:58:30.927354 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.927312    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s64mf" podStartSLOduration=9.303857682 podStartE2EDuration="37.927297842s" totalImagesPullingTime="28.62344016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:30.926139879 +0000 UTC m=+38.846760286" watchObservedRunningTime="2026-09-10 20:58:30.927297842 +0000 UTC m=+38.847918227"
Sep 10 20:58:30.941106 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:30.941069    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8sx7m" podStartSLOduration=2.437129927 podStartE2EDuration="4.941057846s" totalImagesPullingTime="2.503927919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:30.940649421 +0000 UTC m=+38.861269826" watchObservedRunningTime="2026-09-10 20:58:30.941057846 +0000 UTC m=+38.861678223"
Sep 10 20:58:34.127977 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:34.127935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:34.128348 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:34.128067    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:34.128348 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:34.128078    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:34.128348 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:34.128127    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.128113629 +0000 UTC m=+50.048733993 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:34.229071 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:34.229037    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:34.229224 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:34.229151    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:34.229224 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:34.229200    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.229186873 +0000 UTC m=+50.149807250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:37.169217 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.169183    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tt76q"]
Sep 10 20:58:37.205527 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.205497    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tt76q"]
Sep 10 20:58:37.205691 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.205634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.208127 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.208101    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:58:37.208272 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.208239    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:58:37.208420 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.208403    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:58:37.209328 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.209307    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:58:37.209433 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.209342    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2prl\""
Sep 10 20:58:37.352471 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.352437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phgn\" (UniqueName: \"kubernetes.io/projected/69441a4d-87bf-4778-a131-f1058b1d0163-kube-api-access-6phgn\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.352471 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.352475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69441a4d-87bf-4778-a131-f1058b1d0163-crio-socket\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.352676 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.352520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69441a4d-87bf-4778-a131-f1058b1d0163-data-volume\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.352676 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.352633    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.352741 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.352674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.453659 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phgn\" (UniqueName: \"kubernetes.io/projected/69441a4d-87bf-4778-a131-f1058b1d0163-kube-api-access-6phgn\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.453823 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69441a4d-87bf-4778-a131-f1058b1d0163-crio-socket\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.453870 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69441a4d-87bf-4778-a131-f1058b1d0163-data-volume\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.453904 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.453942 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453904    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.454011 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.453992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69441a4d-87bf-4778-a131-f1058b1d0163-crio-socket\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.454011 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.454006    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:37.454111 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.454036    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:37.454111 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.454068    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.954052875 +0000 UTC m=+45.874673248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:37.454198 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.454111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.95409171 +0000 UTC m=+45.874712089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:37.454240 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.454214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69441a4d-87bf-4778-a131-f1058b1d0163-data-volume\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.463253 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.463232    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phgn\" (UniqueName: \"kubernetes.io/projected/69441a4d-87bf-4778-a131-f1058b1d0163-kube-api-access-6phgn\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.957240 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.957206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.957395 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:37.957253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:37.957395 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.957359    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:37.957395 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.957369    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:37.957498 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.957428    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:38.957407891 +0000 UTC m=+46.878028271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:37.957498 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:37.957450    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:38.957439856 +0000 UTC m=+46.878060220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:38.964283 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:38.964245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:38.964684 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:38.964293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:38.964684 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:38.964400    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:38.964684 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:38.964465    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.964450249 +0000 UTC m=+48.885070613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:38.964684 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:38.964401    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:38.964684 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:38.964554    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.964535616 +0000 UTC m=+48.885155998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:39.417457 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.417384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rjnvg"]
Sep 10 20:58:39.439931 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.439906    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rjnvg"]
Sep 10 20:58:39.440064 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.440003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.442957 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.442936    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 20:58:39.443971 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.443950    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 20:58:39.444034 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.443966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 20:58:39.444034 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.443998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7b2cf\""
Sep 10 20:58:39.444034 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.444013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 20:58:39.569184 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.569150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-key\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.569335 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.569203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-cabundle\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.569335 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.569231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdwt\" (UniqueName: \"kubernetes.io/projected/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-kube-api-access-zcdwt\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.670554 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.670481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-cabundle\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.670554 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.670515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdwt\" (UniqueName: \"kubernetes.io/projected/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-kube-api-access-zcdwt\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.670716 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.670690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-key\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.671133 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.671117    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-cabundle\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.673648 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.673632    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-signing-key\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.677959 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.677938    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdwt\" (UniqueName: \"kubernetes.io/projected/0f8a1ae7-b4bb-42ab-af70-b7847ca338d2-kube-api-access-zcdwt\") pod \"service-ca-568869fdc5-rjnvg\" (UID: \"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2\") " pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.748861 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.748833    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rjnvg"
Sep 10 20:58:39.929482 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:39.929417    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hdmm9_a2e81c4a-6431-443e-b0ae-2666493afbd2/dns-node-resolver/0.log"
Sep 10 20:58:40.032164 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:40.032134    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rjnvg"]
Sep 10 20:58:40.035306 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:40.035281    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8a1ae7_b4bb_42ab_af70_b7847ca338d2.slice/crio-0403f37b1eb4c7e4745e2c5f2649d3b3217cdfe2021ab63de9d0d4dba5814dc5 WatchSource:0}: Error finding container 0403f37b1eb4c7e4745e2c5f2649d3b3217cdfe2021ab63de9d0d4dba5814dc5: Status 404 returned error can't find the container with id 0403f37b1eb4c7e4745e2c5f2649d3b3217cdfe2021ab63de9d0d4dba5814dc5
Sep 10 20:58:40.933083 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:40.933041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rjnvg" event={"ID":"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2","Type":"ContainerStarted","Data":"0403f37b1eb4c7e4745e2c5f2649d3b3217cdfe2021ab63de9d0d4dba5814dc5"}
Sep 10 20:58:40.980418 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:40.980379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:40.980573 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:40.980436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:40.980573 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:40.980547    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:40.980665 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:40.980574    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:40.980665 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:40.980648    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.980628753 +0000 UTC m=+52.901249137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:40.980753 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:40.980668    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.980658878 +0000 UTC m=+52.901279243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:41.129744 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:41.129714    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zp29k_2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f/node-ca/0.log"
Sep 10 20:58:41.729560 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:41.729498    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8sx7m_f18c590a-d095-44c2-b581-69fb5f7f00cf/serve-healthcheck-canary/0.log"
Sep 10 20:58:42.192123 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:42.192098    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:42.192448 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:42.192237    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:58:42.192448 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:42.192251    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d8d7559df-fwcz6: secret "image-registry-tls" not found
Sep 10 20:58:42.192448 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:42.192300    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls podName:d239eefe-d6a3-43fe-953a-6ead4d6cd3ba nodeName:}" failed. No retries permitted until 2026-09-10 20:58:58.192287471 +0000 UTC m=+66.112907839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls") pod "image-registry-d8d7559df-fwcz6" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba") : secret "image-registry-tls" not found
Sep 10 20:58:42.292965 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:42.292933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:42.293113 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:42.293046    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:42.293153 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:42.293113    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls podName:2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:58.293094608 +0000 UTC m=+66.213714975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls") pod "dns-default-q996r" (UID: "2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:42.940138 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:42.940101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rjnvg" event={"ID":"0f8a1ae7-b4bb-42ab-af70-b7847ca338d2","Type":"ContainerStarted","Data":"2f49b3c52380cad3d66b1e8d12a2e6b0d18bcc52cf1b9bd7ac77bbdd1454964e"}
Sep 10 20:58:42.954977 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:42.954933    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rjnvg" podStartSLOduration=1.496307685 podStartE2EDuration="3.954918428s" totalImagesPullingTime="2.458610743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:42.954347945 +0000 UTC m=+50.874968349" watchObservedRunningTime="2026-09-10 20:58:42.954918428 +0000 UTC m=+50.875538822"
Sep 10 20:58:45.015881 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:45.015846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:45.016415 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:45.015987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:45.016415 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:45.016111    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.016415 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:58:45.016182    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls podName:69441a4d-87bf-4778-a131-f1058b1d0163 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:53.016161667 +0000 UTC m=+60.936782048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tt76q" (UID: "69441a4d-87bf-4778-a131-f1058b1d0163") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.016585 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:45.016451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69441a4d-87bf-4778-a131-f1058b1d0163-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:50.872144 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:50.872108    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zz5x5"
Sep 10 20:58:53.077393 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.077343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:53.079585 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.079550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69441a4d-87bf-4778-a131-f1058b1d0163-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tt76q\" (UID: \"69441a4d-87bf-4778-a131-f1058b1d0163\") " pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:53.117116 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.117088    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x2prl\""
Sep 10 20:58:53.125447 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.125415    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tt76q"
Sep 10 20:58:53.391933 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.391852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tt76q"]
Sep 10 20:58:53.394642 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:53.394608    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69441a4d_87bf_4778_a131_f1058b1d0163.slice/crio-9513d2f162d51d59b15ac31058e82f7cdf12b6bbd387c69eb441494c4b1fe7fa WatchSource:0}: Error finding container 9513d2f162d51d59b15ac31058e82f7cdf12b6bbd387c69eb441494c4b1fe7fa: Status 404 returned error can't find the container with id 9513d2f162d51d59b15ac31058e82f7cdf12b6bbd387c69eb441494c4b1fe7fa
Sep 10 20:58:53.968362 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.968278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tt76q" event={"ID":"69441a4d-87bf-4778-a131-f1058b1d0163","Type":"ContainerStarted","Data":"64ed73f979946b3724cf5303427faee778efba95251625e1ce9df9263ec8844f"}
Sep 10 20:58:53.968362 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:53.968334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tt76q" event={"ID":"69441a4d-87bf-4778-a131-f1058b1d0163","Type":"ContainerStarted","Data":"9513d2f162d51d59b15ac31058e82f7cdf12b6bbd387c69eb441494c4b1fe7fa"}
Sep 10 20:58:54.972672 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:54.972642    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tt76q" event={"ID":"69441a4d-87bf-4778-a131-f1058b1d0163","Type":"ContainerStarted","Data":"4540b6bb4501cb90bd6d257ef954b61479c791ed4eb28c2191beeaad2adb255b"}
Sep 10 20:58:55.977437 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:55.977401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tt76q" event={"ID":"69441a4d-87bf-4778-a131-f1058b1d0163","Type":"ContainerStarted","Data":"771fe4c8aa9cfdd645b710301bfe7df12393d10c939a6f9a126f338868611107"}
Sep 10 20:58:58.215738 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.215694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:58.218005 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.217981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"image-registry-d8d7559df-fwcz6\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") " pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:58.316924 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.316891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:58.317080 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.316934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:58.317080 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.316964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:58.319085 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.319065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a-metrics-tls\") pod \"dns-default-q996r\" (UID: \"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a\") " pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:58.319592 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.319576    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:58.319666 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.319610    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:58.329588 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.329558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca93feb-5081-4d12-ba13-def3a077f69c-original-pull-secret\") pod \"global-pull-secret-syncer-cvk8k\" (UID: \"eca93feb-5081-4d12-ba13-def3a077f69c\") " pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:58.329588 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.329558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d82ca890-8cb0-42e1-9325-555f07de811d-metrics-certs\") pod \"network-metrics-daemon-9xrp9\" (UID: \"d82ca890-8cb0-42e1-9325-555f07de811d\") " pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:58.418256 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.418212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:58.420547 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.420529    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:58.430869 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.430850    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:58.434962 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.434948    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g6p7f\""
Sep 10 20:58:58.441248 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.441227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbh28\" (UniqueName: \"kubernetes.io/projected/e6bca96e-8693-433d-bec4-e7345926321f-kube-api-access-dbh28\") pod \"network-check-target-zn4gx\" (UID: \"e6bca96e-8693-433d-bec4-e7345926321f\") " pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:58.442939 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.442925    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9xrp9"
Sep 10 20:58:58.509540 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.509516    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-88l8n\""
Sep 10 20:58:58.517140 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.516916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:58.522467 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.522446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvk8k"
Sep 10 20:58:58.532266 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.532244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-56bmz\""
Sep 10 20:58:58.539603 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.539580    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q996r"
Sep 10 20:58:58.729454 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.729395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pbskf\""
Sep 10 20:58:58.734323 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.734277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tt76q" podStartSLOduration=19.870410778 podStartE2EDuration="21.734260099s" totalImagesPullingTime="1.863849321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.995349909 +0000 UTC m=+63.915970316" watchObservedRunningTime="2026-09-10 20:58:58.734260099 +0000 UTC m=+66.654880485"
Sep 10 20:58:58.734446 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.734433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9xrp9"]
Sep 10 20:58:58.737357 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.737338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:58:58.748225 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:58.748199    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82ca890_8cb0_42e1_9325_555f07de811d.slice/crio-0e6e7cffa5b60b3e333111b3645939cb467afd9ab539e2b4b2b439714400a141 WatchSource:0}: Error finding container 0e6e7cffa5b60b3e333111b3645939cb467afd9ab539e2b4b2b439714400a141: Status 404 returned error can't find the container with id 0e6e7cffa5b60b3e333111b3645939cb467afd9ab539e2b4b2b439714400a141
Sep 10 20:58:58.808366 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.808332    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:58:58.813955 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:58.813923    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd239eefe_d6a3_43fe_953a_6ead4d6cd3ba.slice/crio-20bc471364e253a3cb58219d0f77ba1873dda3f02726c5fe8331f6fc66de3737 WatchSource:0}: Error finding container 20bc471364e253a3cb58219d0f77ba1873dda3f02726c5fe8331f6fc66de3737: Status 404 returned error can't find the container with id 20bc471364e253a3cb58219d0f77ba1873dda3f02726c5fe8331f6fc66de3737
Sep 10 20:58:58.827900 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.827874    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvk8k"]
Sep 10 20:58:58.831226 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:58.831199    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca93feb_5081_4d12_ba13_def3a077f69c.slice/crio-c477980a4eed4351c1fb5a9682f9595fe7d7209d818cbfa4d0aea5369d8a15ca WatchSource:0}: Error finding container c477980a4eed4351c1fb5a9682f9595fe7d7209d818cbfa4d0aea5369d8a15ca: Status 404 returned error can't find the container with id c477980a4eed4351c1fb5a9682f9595fe7d7209d818cbfa4d0aea5369d8a15ca
Sep 10 20:58:58.855477 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.855454    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q996r"]
Sep 10 20:58:58.860023 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:58.859992    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0342b9_8b2d_4baf_b70f_c71bf72b8f7a.slice/crio-15d2ed441a84a1f79a42be3a12e84dcf2d3b9f30cb08621a455bdf41f771650f WatchSource:0}: Error finding container 15d2ed441a84a1f79a42be3a12e84dcf2d3b9f30cb08621a455bdf41f771650f: Status 404 returned error can't find the container with id 15d2ed441a84a1f79a42be3a12e84dcf2d3b9f30cb08621a455bdf41f771650f
Sep 10 20:58:58.990241 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.990156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" event={"ID":"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba","Type":"ContainerStarted","Data":"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"}
Sep 10 20:58:58.990241 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.990213    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" event={"ID":"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba","Type":"ContainerStarted","Data":"20bc471364e253a3cb58219d0f77ba1873dda3f02726c5fe8331f6fc66de3737"}
Sep 10 20:58:58.990430 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.990278    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:58:58.991403 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.991381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xrp9" event={"ID":"d82ca890-8cb0-42e1-9325-555f07de811d","Type":"ContainerStarted","Data":"0e6e7cffa5b60b3e333111b3645939cb467afd9ab539e2b4b2b439714400a141"}
Sep 10 20:58:58.992435 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.992414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q996r" event={"ID":"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a","Type":"ContainerStarted","Data":"15d2ed441a84a1f79a42be3a12e84dcf2d3b9f30cb08621a455bdf41f771650f"}
Sep 10 20:58:58.993486 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:58.993467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvk8k" event={"ID":"eca93feb-5081-4d12-ba13-def3a077f69c","Type":"ContainerStarted","Data":"c477980a4eed4351c1fb5a9682f9595fe7d7209d818cbfa4d0aea5369d8a15ca"}
Sep 10 20:58:59.008643 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:59.008541    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" podStartSLOduration=44.008525564 podStartE2EDuration="44.008525564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:59.008112106 +0000 UTC m=+66.928732515" watchObservedRunningTime="2026-09-10 20:58:59.008525564 +0000 UTC m=+66.929145951"
Sep 10 20:58:59.030897 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:58:59.030876    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zn4gx"]
Sep 10 20:58:59.034420 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:58:59.034394    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bca96e_8693_433d_bec4_e7345926321f.slice/crio-cacd9fb9159abdb3a1caef9a478b8a085e6ab3a3af10023f7c6c31cd04a947c3 WatchSource:0}: Error finding container cacd9fb9159abdb3a1caef9a478b8a085e6ab3a3af10023f7c6c31cd04a947c3: Status 404 returned error can't find the container with id cacd9fb9159abdb3a1caef9a478b8a085e6ab3a3af10023f7c6c31cd04a947c3
Sep 10 20:59:00.001621 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:00.001577    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn4gx" event={"ID":"e6bca96e-8693-433d-bec4-e7345926321f","Type":"ContainerStarted","Data":"cacd9fb9159abdb3a1caef9a478b8a085e6ab3a3af10023f7c6c31cd04a947c3"}
Sep 10 20:59:01.196475 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.196421    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"]
Sep 10 20:59:01.207830 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.201672    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:59:01.207830 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.202039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.207830 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.202437    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"]
Sep 10 20:59:01.207830 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.205852    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.208782 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.208500    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:59:01.212203 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.210865    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:59:01.212203 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.211302    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 20:59:01.212203 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.211512    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 20:59:01.212203 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.211688    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 20:59:01.214215 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.212516    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 20:59:01.214215 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.212731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:59:01.216258 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.216236    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 20:59:01.226200 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.223480    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"]
Sep 10 20:59:01.226200 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.223531    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"]
Sep 10 20:59:01.239700 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.239700 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-tmp\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.239909 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4kvf\" (UniqueName: \"kubernetes.io/projected/e7458ac9-7131-4bad-967f-b10f39339f76-kube-api-access-s4kvf\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.239909 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58nsr\" (UniqueName: \"kubernetes.io/projected/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-kube-api-access-58nsr\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.239909 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.239909 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7458ac9-7131-4bad-967f-b10f39339f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.240112 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.240112 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-klusterlet-config\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.240112 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.239970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340561    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-tmp\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4kvf\" (UniqueName: \"kubernetes.io/projected/e7458ac9-7131-4bad-967f-b10f39339f76-kube-api-access-s4kvf\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58nsr\" (UniqueName: \"kubernetes.io/projected/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-kube-api-access-58nsr\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7458ac9-7131-4bad-967f-b10f39339f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.340836 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-klusterlet-config\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.341300 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.341090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-tmp\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.341895 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.340816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.341895 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.341522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.341895 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.341706    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7458ac9-7131-4bad-967f-b10f39339f76-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.346274 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.346218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.349135 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.348965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.350556 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.350531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-klusterlet-config\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.351851 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.351790    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-ca\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.352015 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.351992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e7458ac9-7131-4bad-967f-b10f39339f76-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.362225 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.362128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4kvf\" (UniqueName: \"kubernetes.io/projected/e7458ac9-7131-4bad-967f-b10f39339f76-kube-api-access-s4kvf\") pod \"cluster-proxy-proxy-agent-6d794bf49-2k8lr\" (UID: \"e7458ac9-7131-4bad-967f-b10f39339f76\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.362338 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.362319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58nsr\" (UniqueName: \"kubernetes.io/projected/5a1ed60e-fcdf-4fc3-a901-ec998d87e834-kube-api-access-58nsr\") pod \"klusterlet-addon-workmgr-679f78fb97-l5r6c\" (UID: \"5a1ed60e-fcdf-4fc3-a901-ec998d87e834\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.518674 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.518639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:01.537856 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.537825    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"
Sep 10 20:59:01.799844 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.799817    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"]
Sep 10 20:59:01.828398 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:01.828373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr"]
Sep 10 20:59:02.013552 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.013516    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q996r" event={"ID":"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a","Type":"ContainerStarted","Data":"a5131c58ae0e3680e9863e7f6baf31cb287e9c56cbcdc8295e5290c99abc9530"}
Sep 10 20:59:02.013731 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.013581    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q996r"
Sep 10 20:59:02.013731 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.013608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q996r" event={"ID":"2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a","Type":"ContainerStarted","Data":"2a562071abe7564378f99def956abffc72e8b04c1850c798b0ec6f6a449bdfd7"}
Sep 10 20:59:02.015691 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.015665    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xrp9" event={"ID":"d82ca890-8cb0-42e1-9325-555f07de811d","Type":"ContainerStarted","Data":"4acbb0dadff0d6a7698cce1e468542c3bdebbc7d837246ded4409bcb40e97c14"}
Sep 10 20:59:02.015820 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.015713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9xrp9" event={"ID":"d82ca890-8cb0-42e1-9325-555f07de811d","Type":"ContainerStarted","Data":"591da4025464de881ce06aa5d3b3e387a9f5c1aedd8ff4a5684b7e99ed46e494"}
Sep 10 20:59:02.031233 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.031191    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q996r" podStartSLOduration=33.7417126 podStartE2EDuration="36.031176922s" totalImagesPullingTime="2.289464322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:02.029476246 +0000 UTC m=+69.950096652" watchObservedRunningTime="2026-09-10 20:59:02.031176922 +0000 UTC m=+69.951797308"
Sep 10 20:59:02.043731 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:02.043680    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9xrp9" podStartSLOduration=66.664550935 podStartE2EDuration="1m9.043662851s" totalImagesPullingTime="2.379111916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:02.043483744 +0000 UTC m=+69.964104149" watchObservedRunningTime="2026-09-10 20:59:02.043662851 +0000 UTC m=+69.964283240"
Sep 10 20:59:02.660354 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:59:02.660319    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7458ac9_7131_4bad_967f_b10f39339f76.slice/crio-a7c580a8408f950aa2e2160655040dc8ed3a7f1b13b1d9cf0c9978d0d87b3991 WatchSource:0}: Error finding container a7c580a8408f950aa2e2160655040dc8ed3a7f1b13b1d9cf0c9978d0d87b3991: Status 404 returned error can't find the container with id a7c580a8408f950aa2e2160655040dc8ed3a7f1b13b1d9cf0c9978d0d87b3991
Sep 10 20:59:03.020698 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:03.020653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr" event={"ID":"e7458ac9-7131-4bad-967f-b10f39339f76","Type":"ContainerStarted","Data":"a7c580a8408f950aa2e2160655040dc8ed3a7f1b13b1d9cf0c9978d0d87b3991"}
Sep 10 20:59:04.013790 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:59:04.013754    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1ed60e_fcdf_4fc3_a901_ec998d87e834.slice/crio-c31b2eb64ab692fe1e2e180373f6f1884e4659d1976cf05c1ad88d11a1ac782e WatchSource:0}: Error finding container c31b2eb64ab692fe1e2e180373f6f1884e4659d1976cf05c1ad88d11a1ac782e: Status 404 returned error can't find the container with id c31b2eb64ab692fe1e2e180373f6f1884e4659d1976cf05c1ad88d11a1ac782e
Sep 10 20:59:04.025153 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:04.025125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c" event={"ID":"5a1ed60e-fcdf-4fc3-a901-ec998d87e834","Type":"ContainerStarted","Data":"c31b2eb64ab692fe1e2e180373f6f1884e4659d1976cf05c1ad88d11a1ac782e"}
Sep 10 20:59:05.034184 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:05.034142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zn4gx" event={"ID":"e6bca96e-8693-433d-bec4-e7345926321f","Type":"ContainerStarted","Data":"f5a380c8e5ef1dfd136547c48f3df7bb3606fdbbd980f362c258b963a4c38a68"}
Sep 10 20:59:05.034976 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:05.034954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:59:05.038193 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:05.038169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvk8k" event={"ID":"eca93feb-5081-4d12-ba13-def3a077f69c","Type":"ContainerStarted","Data":"4ace2c3ec3e92014b522595f94ad9730dee58106386baf43bd5011515849ce2d"}
Sep 10 20:59:05.053918 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:05.053864    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zn4gx" podStartSLOduration=67.035847 podStartE2EDuration="1m12.053844943s" totalImagesPullingTime="5.017997943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:05.051886479 +0000 UTC m=+72.972506892" watchObservedRunningTime="2026-09-10 20:59:05.053844943 +0000 UTC m=+72.974465329"
Sep 10 20:59:05.068390 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:05.068334    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cvk8k" podStartSLOduration=66.843955291 podStartE2EDuration="1m12.068314912s" totalImagesPullingTime="5.224359621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:05.067834118 +0000 UTC m=+72.988454531" watchObservedRunningTime="2026-09-10 20:59:05.068314912 +0000 UTC m=+72.988935300"
Sep 10 20:59:10.059098 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.059053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr" event={"ID":"e7458ac9-7131-4bad-967f-b10f39339f76","Type":"ContainerStarted","Data":"e193bf359fd110d3b025852c4ecfb4c9b311cef84384f38aafdd943639385fce"}
Sep 10 20:59:10.059538 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.059111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr" event={"ID":"e7458ac9-7131-4bad-967f-b10f39339f76","Type":"ContainerStarted","Data":"a00a36113519724d128a1fe28fe275a1070598443fcd111f80096d056a581e3e"}
Sep 10 20:59:10.059538 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.059132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr" event={"ID":"e7458ac9-7131-4bad-967f-b10f39339f76","Type":"ContainerStarted","Data":"0e44ff94cb0dde0304fc81300a3762df866af24b77aa20449d251749d5527930"}
Sep 10 20:59:10.060558 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.060529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c" event={"ID":"5a1ed60e-fcdf-4fc3-a901-ec998d87e834","Type":"ContainerStarted","Data":"0617bbdc16ce8dcf1d062743a968fa3132b262d3bd2840bba11d662d4f7129c2"}
Sep 10 20:59:10.060774 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.060756    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:10.062530 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.062507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c"
Sep 10 20:59:10.080388 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.080342    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d794bf49-2k8lr" podStartSLOduration=2.701160141 podStartE2EDuration="9.080328738s" totalImagesPullingTime="6.379168597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:10.078399918 +0000 UTC m=+77.999020319" watchObservedRunningTime="2026-09-10 20:59:10.080328738 +0000 UTC m=+78.000949124"
Sep 10 20:59:10.094390 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:10.094348    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-679f78fb97-l5r6c" podStartSLOduration=4.087915812 podStartE2EDuration="9.094334869s" totalImagesPullingTime="5.006419057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:10.09348895 +0000 UTC m=+78.014109352" watchObservedRunningTime="2026-09-10 20:59:10.094334869 +0000 UTC m=+78.014955257"
Sep 10 20:59:12.023007 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:12.022975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q996r"
Sep 10 20:59:15.964698 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.964574    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5pzbc"]
Sep 10 20:59:15.968978 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.968955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:15.972074 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.972054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:59:15.972437 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.972420    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:59:15.973369 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.973346    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:59:15.973525 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.973350    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:59:15.973525 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.973403    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:59:15.973525 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.973353    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:59:15.973673 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:15.973659    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-46kj6\""
Sep 10 20:59:16.058548 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-root\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058724 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-tls\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058724 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058594    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058724 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-sys\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058724 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zn4\" (UniqueName: \"kubernetes.io/projected/ccfac512-7196-4a29-adeb-1ceef41c63c7-kube-api-access-z8zn4\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058724 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-metrics-client-ca\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058990 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-textfile\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058990 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.058990 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.058817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-wtmp\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159531 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-metrics-client-ca\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-textfile\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-wtmp\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-root\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159704 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-tls\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159698    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-root\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159738    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-wtmp\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-sys\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zn4\" (UniqueName: \"kubernetes.io/projected/ccfac512-7196-4a29-adeb-1ceef41c63c7-kube-api-access-z8zn4\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159931    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccfac512-7196-4a29-adeb-1ceef41c63c7-sys\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.159997 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.159973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-textfile\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.160330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.160184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-metrics-client-ca\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.160330 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.160217    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-accelerators-collector-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.162088 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.162070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-tls\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.162171 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.162094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccfac512-7196-4a29-adeb-1ceef41c63c7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.168370 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.168343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zn4\" (UniqueName: \"kubernetes.io/projected/ccfac512-7196-4a29-adeb-1ceef41c63c7-kube-api-access-z8zn4\") pod \"node-exporter-5pzbc\" (UID: \"ccfac512-7196-4a29-adeb-1ceef41c63c7\") " pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.277968 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:16.277904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5pzbc"
Sep 10 20:59:16.288610 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:59:16.288586    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccfac512_7196_4a29_adeb_1ceef41c63c7.slice/crio-98b560e1e7579247d5b52feb34f7c0e3dd94fd08df64160af787fdca1e33e8c1 WatchSource:0}: Error finding container 98b560e1e7579247d5b52feb34f7c0e3dd94fd08df64160af787fdca1e33e8c1: Status 404 returned error can't find the container with id 98b560e1e7579247d5b52feb34f7c0e3dd94fd08df64160af787fdca1e33e8c1
Sep 10 20:59:17.086778 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:17.086735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzbc" event={"ID":"ccfac512-7196-4a29-adeb-1ceef41c63c7","Type":"ContainerStarted","Data":"98b560e1e7579247d5b52feb34f7c0e3dd94fd08df64160af787fdca1e33e8c1"}
Sep 10 20:59:18.091500 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:18.091466    2562 generic.go:356] "Generic (PLEG): container finished" podID="ccfac512-7196-4a29-adeb-1ceef41c63c7" containerID="10f9fe8f5e1b9d77ec210e540152151a12ec632c6d6fed4c14226595625c91e2" exitCode=0
Sep 10 20:59:18.091898 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:18.091530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzbc" event={"ID":"ccfac512-7196-4a29-adeb-1ceef41c63c7","Type":"ContainerDied","Data":"10f9fe8f5e1b9d77ec210e540152151a12ec632c6d6fed4c14226595625c91e2"}
Sep 10 20:59:19.097770 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:19.097731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzbc" event={"ID":"ccfac512-7196-4a29-adeb-1ceef41c63c7","Type":"ContainerStarted","Data":"dd3686c0f517ab317a01519dadf3c7ba8f540d25ec8c1da034cca41c93fc787f"}
Sep 10 20:59:19.098229 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:19.097793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5pzbc" event={"ID":"ccfac512-7196-4a29-adeb-1ceef41c63c7","Type":"ContainerStarted","Data":"66c986fb46e1850b4b53d094d8ad4815f91d3488253824665ec8aa30cddfd00b"}
Sep 10 20:59:21.208199 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:21.208174    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:59:21.232942 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:21.232891    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5pzbc" podStartSLOduration=5.278294524 podStartE2EDuration="6.232875525s" totalImagesPullingTime="894.286084ms" totalInitContainerRuntime="60.294917ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:19.118761005 +0000 UTC m=+87.039381410" watchObservedRunningTime="2026-09-10 20:59:21.232875525 +0000 UTC m=+89.153495971"
Sep 10 20:59:26.236459 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.236399    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" podUID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" containerName="registry" containerID="cri-o://4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea" gracePeriod=30
Sep 10 20:59:26.470903 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.470880    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:59:26.525484 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525429    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525484 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525639 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525495    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525639 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525519    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhqqw\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525639 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525573    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525639 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525601    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525639 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.525913 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") pod \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\" (UID: \"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba\") "
Sep 10 20:59:26.526031 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.525971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:26.526145 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.526104    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 20:59:26.528221 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.528169    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:26.528338 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.528223    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:26.528338 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.528258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 20:59:26.528338 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.528309    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:26.528500 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.528359    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw" (OuterVolumeSpecName: "kube-api-access-fhqqw") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "kube-api-access-fhqqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 20:59:26.534039 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.534017    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" (UID: "d239eefe-d6a3-43fe-953a-6ead4d6cd3ba"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 20:59:26.627009 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.626989    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-image-registry-private-configuration\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627011    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-bound-sa-token\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627021    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-trusted-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627028    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627036    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-registry-certificates\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627044    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-ca-trust-extracted\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627052    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-installation-pull-secrets\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:26.627089 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:26.627060    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqqw\" (UniqueName: \"kubernetes.io/projected/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba-kube-api-access-fhqqw\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 20:59:27.125834 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.125786    2562 generic.go:356] "Generic (PLEG): container finished" podID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" containerID="4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea" exitCode=0
Sep 10 20:59:27.126000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.125842    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6"
Sep 10 20:59:27.126000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.125865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" event={"ID":"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba","Type":"ContainerDied","Data":"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"}
Sep 10 20:59:27.126000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.125908    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d8d7559df-fwcz6" event={"ID":"d239eefe-d6a3-43fe-953a-6ead4d6cd3ba","Type":"ContainerDied","Data":"20bc471364e253a3cb58219d0f77ba1873dda3f02726c5fe8331f6fc66de3737"}
Sep 10 20:59:27.126000 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.125931    2562 scope.go:117] "RemoveContainer" containerID="4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"
Sep 10 20:59:27.134176 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.134153    2562 scope.go:117] "RemoveContainer" containerID="4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"
Sep 10 20:59:27.134473 ip-10-0-135-61 kubenswrapper[2562]: E0910 20:59:27.134453    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea\": container with ID starting with 4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea not found: ID does not exist" containerID="4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"
Sep 10 20:59:27.134542 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.134483    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea"} err="failed to get container status \"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea\": rpc error: code = NotFound desc = could not find container \"4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea\": container with ID starting with 4198402534fb9703abb61ea6f0e281d4fec39fee48ddb46c7e56e1611ea535ea not found: ID does not exist"
Sep 10 20:59:27.143846 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.143824    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:59:27.147560 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:27.147540    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d8d7559df-fwcz6"]
Sep 10 20:59:28.616707 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:28.616673    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" path="/var/lib/kubelet/pods/d239eefe-d6a3-43fe-953a-6ead4d6cd3ba/volumes"
Sep 10 20:59:35.068796 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.068766    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 20:59:35.069207 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.068993    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" containerName="registry"
Sep 10 20:59:35.069207 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.069005    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" containerName="registry"
Sep 10 20:59:35.069207 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.069053    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d239eefe-d6a3-43fe-953a-6ead4d6cd3ba" containerName="registry"
Sep 10 20:59:35.118652 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.118624    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 20:59:35.118788 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.118712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.121645 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.121613    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 20:59:35.121782 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.121754    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9msmf\""
Sep 10 20:59:35.121880 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.121817    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 20:59:35.121880 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.121868    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 20:59:35.123060 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.123041    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 20:59:35.123139 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.123117    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 20:59:35.123139 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.123125    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 20:59:35.123596 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.123574    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 20:59:35.129777 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.129758    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 20:59:35.185248 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185355 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185355 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185355 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185506 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgt8x\" (UniqueName: \"kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185506 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185385    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.185506 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.185440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286446 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286446 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgt8x\" (UniqueName: \"kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286612 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286612 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286612 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286612 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.286612 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.286552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.287388 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.287362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.287503 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.287480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.287629 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.287602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.287839 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.287800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.289059 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.289042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.289250 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.289231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.294642 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.294620    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgt8x\" (UniqueName: \"kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x\") pod \"console-78fcbdf65d-7pbqp\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") " pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.432444 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.432387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:35.724569 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:35.724539    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 20:59:35.727893 ip-10-0-135-61 kubenswrapper[2562]: W0910 20:59:35.727861    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c772fa_230f_4c22_8460_290c598e50cc.slice/crio-c59007f6ff4bb94325382445f84649f6d04e76f8103a9eeebc3223eb85a75799 WatchSource:0}: Error finding container c59007f6ff4bb94325382445f84649f6d04e76f8103a9eeebc3223eb85a75799: Status 404 returned error can't find the container with id c59007f6ff4bb94325382445f84649f6d04e76f8103a9eeebc3223eb85a75799
Sep 10 20:59:36.157053 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:36.156976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fcbdf65d-7pbqp" event={"ID":"79c772fa-230f-4c22-8460-290c598e50cc","Type":"ContainerStarted","Data":"c59007f6ff4bb94325382445f84649f6d04e76f8103a9eeebc3223eb85a75799"}
Sep 10 20:59:37.048955 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:37.048919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zn4gx"
Sep 10 20:59:39.169324 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:39.169240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fcbdf65d-7pbqp" event={"ID":"79c772fa-230f-4c22-8460-290c598e50cc","Type":"ContainerStarted","Data":"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"}
Sep 10 20:59:39.186141 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:39.186099    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78fcbdf65d-7pbqp" podStartSLOduration=1.095785137 podStartE2EDuration="4.186083502s" totalImagesPullingTime="3.090298365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:39.184858159 +0000 UTC m=+107.105478567" watchObservedRunningTime="2026-09-10 20:59:39.186083502 +0000 UTC m=+107.106703878"
Sep 10 20:59:45.432954 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:45.432890    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:45.432954 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:45.432961    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:45.437454 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:45.437432    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 20:59:46.197970 ip-10-0-135-61 kubenswrapper[2562]: I0910 20:59:46.197940    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 21:00:51.904121 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:00:51.904039    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 21:01:16.922689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:16.922652    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78fcbdf65d-7pbqp" podUID="79c772fa-230f-4c22-8460-290c598e50cc" containerName="console" containerID="cri-o://993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0" gracePeriod=15
Sep 10 21:01:17.156933 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.156909    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 21:01:17.185012 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.184956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgt8x\" (UniqueName: \"kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185012 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.184987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185052    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert\") pod \"79c772fa-230f-4c22-8460-290c598e50cc\" (UID: \"79c772fa-230f-4c22-8460-290c598e50cc\") "
Sep 10 21:01:17.185565 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca" (OuterVolumeSpecName: "service-ca") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:17.185565 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config" (OuterVolumeSpecName: "console-config") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:17.185715 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185582    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:17.185715 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.185607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:17.187270 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.187237    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:17.187365 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.187277    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:17.187365 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.187306    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x" (OuterVolumeSpecName: "kube-api-access-fgt8x") pod "79c772fa-230f-4c22-8460-290c598e50cc" (UID: "79c772fa-230f-4c22-8460-290c598e50cc"). InnerVolumeSpecName "kube-api-access-fgt8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:01:17.285914 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285875    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgt8x\" (UniqueName: \"kubernetes.io/projected/79c772fa-230f-4c22-8460-290c598e50cc-kube-api-access-fgt8x\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.285914 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285907    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-trusted-ca-bundle\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.285914 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285916    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-serving-cert\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.285914 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285924    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-console-config\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.286163 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285933    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.286163 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285941    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79c772fa-230f-4c22-8460-290c598e50cc-console-oauth-config\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.286163 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.285949    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79c772fa-230f-4c22-8460-290c598e50cc-oauth-serving-cert\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:01:17.496200 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.496163    2562 generic.go:356] "Generic (PLEG): container finished" podID="79c772fa-230f-4c22-8460-290c598e50cc" containerID="993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0" exitCode=0
Sep 10 21:01:17.496377 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.496205    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78fcbdf65d-7pbqp"
Sep 10 21:01:17.496377 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.496241    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fcbdf65d-7pbqp" event={"ID":"79c772fa-230f-4c22-8460-290c598e50cc","Type":"ContainerDied","Data":"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"}
Sep 10 21:01:17.496377 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.496311    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78fcbdf65d-7pbqp" event={"ID":"79c772fa-230f-4c22-8460-290c598e50cc","Type":"ContainerDied","Data":"c59007f6ff4bb94325382445f84649f6d04e76f8103a9eeebc3223eb85a75799"}
Sep 10 21:01:17.496377 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.496337    2562 scope.go:117] "RemoveContainer" containerID="993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"
Sep 10 21:01:17.504782 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.504766    2562 scope.go:117] "RemoveContainer" containerID="993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"
Sep 10 21:01:17.505080 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:01:17.505058    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0\": container with ID starting with 993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0 not found: ID does not exist" containerID="993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"
Sep 10 21:01:17.505156 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.505091    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0"} err="failed to get container status \"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0\": rpc error: code = NotFound desc = could not find container \"993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0\": container with ID starting with 993a4005ec89a66ec5d0724b4a8dbc3366d6e28a7bb010bf9ef27adefd118cb0 not found: ID does not exist"
Sep 10 21:01:17.515087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.515062    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 21:01:17.517991 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:17.517972    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78fcbdf65d-7pbqp"]
Sep 10 21:01:18.616330 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:18.616297    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c772fa-230f-4c22-8460-290c598e50cc" path="/var/lib/kubelet/pods/79c772fa-230f-4c22-8460-290c598e50cc/volumes"
Sep 10 21:01:41.399136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.399101    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"]
Sep 10 21:01:41.399595 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.399420    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c772fa-230f-4c22-8460-290c598e50cc" containerName="console"
Sep 10 21:01:41.399595 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.399436    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c772fa-230f-4c22-8460-290c598e50cc" containerName="console"
Sep 10 21:01:41.399595 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.399498    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c772fa-230f-4c22-8460-290c598e50cc" containerName="console"
Sep 10 21:01:41.403577 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.403557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.406402 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.406377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:01:41.406402 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.406384    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 21:01:41.406551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.406412    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vbt4x\""
Sep 10 21:01:41.414661 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.414632    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"]
Sep 10 21:01:41.447675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.447640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.447675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.447680    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqc4\" (UniqueName: \"kubernetes.io/projected/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-kube-api-access-9sqc4\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.548290 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.548257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.548449 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.548295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqc4\" (UniqueName: \"kubernetes.io/projected/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-kube-api-access-9sqc4\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.548638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.548616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.557357 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.557324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqc4\" (UniqueName: \"kubernetes.io/projected/962bbf3d-7d09-4ec3-8e71-4d850f792f3c-kube-api-access-9sqc4\") pod \"cert-manager-operator-controller-manager-55c444bf86-m58kv\" (UID: \"962bbf3d-7d09-4ec3-8e71-4d850f792f3c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.712635 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.712602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"
Sep 10 21:01:41.984837 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:41.984798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv"]
Sep 10 21:01:41.987663 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:01:41.987639    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962bbf3d_7d09_4ec3_8e71_4d850f792f3c.slice/crio-2ed0ae87ebec5207aaaaa60cd45fe51e5f7957ac6d218f0cab6c19e3a45363c1 WatchSource:0}: Error finding container 2ed0ae87ebec5207aaaaa60cd45fe51e5f7957ac6d218f0cab6c19e3a45363c1: Status 404 returned error can't find the container with id 2ed0ae87ebec5207aaaaa60cd45fe51e5f7957ac6d218f0cab6c19e3a45363c1
Sep 10 21:01:42.578629 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:42.578593    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv" event={"ID":"962bbf3d-7d09-4ec3-8e71-4d850f792f3c","Type":"ContainerStarted","Data":"2ed0ae87ebec5207aaaaa60cd45fe51e5f7957ac6d218f0cab6c19e3a45363c1"}
Sep 10 21:01:45.590987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:45.590955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv" event={"ID":"962bbf3d-7d09-4ec3-8e71-4d850f792f3c","Type":"ContainerStarted","Data":"8b89a0c719ed1e8b7028247de809e3a5c93f7136ad469c770ff7b13629c204d7"}
Sep 10 21:01:45.611924 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:45.611881    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m58kv" podStartSLOduration=2.034937761 podStartE2EDuration="4.611867149s" totalImagesPullingTime="2.576929388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:01:45.610319669 +0000 UTC m=+233.530940073" watchObservedRunningTime="2026-09-10 21:01:45.611867149 +0000 UTC m=+233.532487534"
Sep 10 21:01:56.051064 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.051028    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2h94d"]
Sep 10 21:01:56.057417 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.057391    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.060252 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.060226    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 21:01:56.061458 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.061432    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cn8zt\""
Sep 10 21:01:56.061600 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.061442    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 21:01:56.063192 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.063173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2h94d"]
Sep 10 21:01:56.148765 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.148728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-bound-sa-token\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.148949 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.148775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47lhk\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-kube-api-access-47lhk\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.249709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.249667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-bound-sa-token\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.249911 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.249731    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47lhk\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-kube-api-access-47lhk\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.257602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.257578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-bound-sa-token\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.257761 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.257744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47lhk\" (UniqueName: \"kubernetes.io/projected/27f8a2a5-6936-4034-a5a0-167fd22ceb33-kube-api-access-47lhk\") pod \"cert-manager-55b4d954d5-2h94d\" (UID: \"27f8a2a5-6936-4034-a5a0-167fd22ceb33\") " pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.366661 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.366552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2h94d"
Sep 10 21:01:56.636398 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:56.636323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2h94d"]
Sep 10 21:01:56.640163 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:01:56.640137    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27f8a2a5_6936_4034_a5a0_167fd22ceb33.slice/crio-708e58075278fdff4a1a3694cb2b0f37c18542f66cfc3c22502b8b60e99c9cb2 WatchSource:0}: Error finding container 708e58075278fdff4a1a3694cb2b0f37c18542f66cfc3c22502b8b60e99c9cb2: Status 404 returned error can't find the container with id 708e58075278fdff4a1a3694cb2b0f37c18542f66cfc3c22502b8b60e99c9cb2
Sep 10 21:01:57.630194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:57.630148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2h94d" event={"ID":"27f8a2a5-6936-4034-a5a0-167fd22ceb33","Type":"ContainerStarted","Data":"708e58075278fdff4a1a3694cb2b0f37c18542f66cfc3c22502b8b60e99c9cb2"}
Sep 10 21:01:59.639021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:59.638932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2h94d" event={"ID":"27f8a2a5-6936-4034-a5a0-167fd22ceb33","Type":"ContainerStarted","Data":"b8d34f0ce4fb743f916e0612999f7690160c810ca72d7042b7c9730e29a02850"}
Sep 10 21:01:59.655577 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:01:59.655530    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2h94d" podStartSLOduration=0.989931342 podStartE2EDuration="3.65551776s" totalImagesPullingTime="2.665586418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:01:59.654326607 +0000 UTC m=+247.574946994" watchObservedRunningTime="2026-09-10 21:01:59.65551776 +0000 UTC m=+247.576138146"
Sep 10 21:02:18.618197 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.618098    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"]
Sep 10 21:02:18.621119 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.621100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.624199 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.624181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:02:18.624412 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.624381    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:02:18.624517 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.624471    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pmkvq\""
Sep 10 21:02:18.624517 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.624471    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:02:18.624634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.624512    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:02:18.635590 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.635566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"]
Sep 10 21:02:18.707648 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.707619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.707843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.707659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-tmp\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.707843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.707677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.707843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.707735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-manifests\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.707843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.707767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftds4\" (UniqueName: \"kubernetes.io/projected/1c5265dc-75d5-430b-934b-91c6572aebf3-kube-api-access-ftds4\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.808911 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.808865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftds4\" (UniqueName: \"kubernetes.io/projected/1c5265dc-75d5-430b-934b-91c6572aebf3-kube-api-access-ftds4\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.809096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.808952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.809096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.808993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-tmp\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.809096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.809047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.809096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.809090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-manifests\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.809468 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.809448    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-manifests\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.811337 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.811316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5265dc-75d5-430b-934b-91c6572aebf3-tmp\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.811620 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.811604    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-webhook-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.811678 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.811658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c5265dc-75d5-430b-934b-91c6572aebf3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.822313 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.822290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftds4\" (UniqueName: \"kubernetes.io/projected/1c5265dc-75d5-430b-934b-91c6572aebf3-kube-api-access-ftds4\") pod \"opendatahub-operator-controller-manager-d855cccc6-nln6b\" (UID: \"1c5265dc-75d5-430b-934b-91c6572aebf3\") " pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:18.931147 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:18.931067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:19.206074 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:19.206044    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"]
Sep 10 21:02:19.209496 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:02:19.209468    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5265dc_75d5_430b_934b_91c6572aebf3.slice/crio-9191b73b2cec4db0941b3614b1b547bc2c5787a6347083694d7f0a2bea979e08 WatchSource:0}: Error finding container 9191b73b2cec4db0941b3614b1b547bc2c5787a6347083694d7f0a2bea979e08: Status 404 returned error can't find the container with id 9191b73b2cec4db0941b3614b1b547bc2c5787a6347083694d7f0a2bea979e08
Sep 10 21:02:19.705573 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:19.705539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b" event={"ID":"1c5265dc-75d5-430b-934b-91c6572aebf3","Type":"ContainerStarted","Data":"9191b73b2cec4db0941b3614b1b547bc2c5787a6347083694d7f0a2bea979e08"}
Sep 10 21:02:22.719757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:22.719720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b" event={"ID":"1c5265dc-75d5-430b-934b-91c6572aebf3","Type":"ContainerStarted","Data":"03d080a3548efd09a9872fcd34d752dd6b1defd881339ddb37bfe3ec6a737fe6"}
Sep 10 21:02:23.724994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:23.724957    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c5265dc-75d5-430b-934b-91c6572aebf3" containerID="03d080a3548efd09a9872fcd34d752dd6b1defd881339ddb37bfe3ec6a737fe6" exitCode=0
Sep 10 21:02:23.725405 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:23.725042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b" event={"ID":"1c5265dc-75d5-430b-934b-91c6572aebf3","Type":"ContainerDied","Data":"03d080a3548efd09a9872fcd34d752dd6b1defd881339ddb37bfe3ec6a737fe6"}
Sep 10 21:02:24.730816 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:24.730776    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b" event={"ID":"1c5265dc-75d5-430b-934b-91c6572aebf3","Type":"ContainerStarted","Data":"77d83ba8e34c81c9d1a11d4216bcc7b448d8e5b353915f662491c0d3dd789aa5"}
Sep 10 21:02:24.731196 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:24.730894    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:24.758064 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:24.758004    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b" podStartSLOduration=2.402083908 podStartE2EDuration="6.757982762s" totalImagesPullingTime="3.317550485s" totalInitContainerRuntime="1.038348369s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:24.756446705 +0000 UTC m=+272.677067120" watchObservedRunningTime="2026-09-10 21:02:24.757982762 +0000 UTC m=+272.678603149"
Sep 10 21:02:35.738300 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:35.738261    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d855cccc6-nln6b"
Sep 10 21:02:38.219014 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.218982    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"]
Sep 10 21:02:38.224835 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.224793    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.228418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.228392    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:02:38.229795 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.229775    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:02:38.229795 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.229789    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:02:38.230430 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.230413    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:02:38.230539 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.230431    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:02:38.230739 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.230722    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hv8md\""
Sep 10 21:02:38.235282 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.235259    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"]
Sep 10 21:02:38.312753 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.312721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-767744946f-t79lq"]
Sep 10 21:02:38.315153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.315125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.317738 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.317706    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 21:02:38.317886 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.317831    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 21:02:38.317981 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.317961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g5wjx\""
Sep 10 21:02:38.318104 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.318078    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 21:02:38.318379 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.318360    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 21:02:38.325748 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.325725    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767744946f-t79lq"]
Sep 10 21:02:38.358648 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.358614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.358648 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.358650    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28vf\" (UniqueName: \"kubernetes.io/projected/708dbef9-0a27-4230-a958-a73588f15671-kube-api-access-q28vf\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.358882 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.358693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-metrics-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.358882 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.358724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/708dbef9-0a27-4230-a958-a73588f15671-manager-config\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.459120 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459081    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.459120 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28vf\" (UniqueName: \"kubernetes.io/projected/708dbef9-0a27-4230-a958-a73588f15671-kube-api-access-q28vf\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.459360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20403f47-03f1-418b-88e6-24884383c103-tls-certs\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.459360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xbvv\" (UniqueName: \"kubernetes.io/projected/20403f47-03f1-418b-88e6-24884383c103-kube-api-access-6xbvv\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.459360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-metrics-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.459360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/708dbef9-0a27-4230-a958-a73588f15671-manager-config\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.459360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20403f47-03f1-418b-88e6-24884383c103-tmp\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.459992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.459969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/708dbef9-0a27-4230-a958-a73588f15671-manager-config\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.461868 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.461843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.461978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.461963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/708dbef9-0a27-4230-a958-a73588f15671-metrics-cert\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.466857 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.466836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28vf\" (UniqueName: \"kubernetes.io/projected/708dbef9-0a27-4230-a958-a73588f15671-kube-api-access-q28vf\") pod \"lws-controller-manager-6f8c8876d7-jxlnl\" (UID: \"708dbef9-0a27-4230-a958-a73588f15671\") " pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.534393 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.534318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:38.559765 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.559729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20403f47-03f1-418b-88e6-24884383c103-tls-certs\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.559941 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.559786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xbvv\" (UniqueName: \"kubernetes.io/projected/20403f47-03f1-418b-88e6-24884383c103-kube-api-access-6xbvv\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.560001 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.559955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20403f47-03f1-418b-88e6-24884383c103-tmp\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.562161 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.562135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20403f47-03f1-418b-88e6-24884383c103-tmp\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.562611 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.562590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20403f47-03f1-418b-88e6-24884383c103-tls-certs\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.567880 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.567855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xbvv\" (UniqueName: \"kubernetes.io/projected/20403f47-03f1-418b-88e6-24884383c103-kube-api-access-6xbvv\") pod \"kube-auth-proxy-767744946f-t79lq\" (UID: \"20403f47-03f1-418b-88e6-24884383c103\") " pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.627778 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.627751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq"
Sep 10 21:02:38.804196 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.804169    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"]
Sep 10 21:02:38.806454 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:02:38.806424    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708dbef9_0a27_4230_a958_a73588f15671.slice/crio-957ca830aef589ad533f9675c4e7983ffa1f926d7d18c241a11655ad58a3141f WatchSource:0}: Error finding container 957ca830aef589ad533f9675c4e7983ffa1f926d7d18c241a11655ad58a3141f: Status 404 returned error can't find the container with id 957ca830aef589ad533f9675c4e7983ffa1f926d7d18c241a11655ad58a3141f
Sep 10 21:02:38.898232 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:38.898210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767744946f-t79lq"]
Sep 10 21:02:38.900322 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:02:38.900296    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20403f47_03f1_418b_88e6_24884383c103.slice/crio-37756e7625174f6191717c7fd1a22e79591f2dfa7a59226a372d4413dd673f94 WatchSource:0}: Error finding container 37756e7625174f6191717c7fd1a22e79591f2dfa7a59226a372d4413dd673f94: Status 404 returned error can't find the container with id 37756e7625174f6191717c7fd1a22e79591f2dfa7a59226a372d4413dd673f94
Sep 10 21:02:39.794104 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:39.794014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq" event={"ID":"20403f47-03f1-418b-88e6-24884383c103","Type":"ContainerStarted","Data":"37756e7625174f6191717c7fd1a22e79591f2dfa7a59226a372d4413dd673f94"}
Sep 10 21:02:39.796385 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:39.796347    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl" event={"ID":"708dbef9-0a27-4230-a958-a73588f15671","Type":"ContainerStarted","Data":"957ca830aef589ad533f9675c4e7983ffa1f926d7d18c241a11655ad58a3141f"}
Sep 10 21:02:42.810825 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:42.810768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq" event={"ID":"20403f47-03f1-418b-88e6-24884383c103","Type":"ContainerStarted","Data":"b9dc5f5f51302251dd3731720b3e23a348275f1a719af711ccf0ed9966eecaf8"}
Sep 10 21:02:42.812764 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:42.812736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl" event={"ID":"708dbef9-0a27-4230-a958-a73588f15671","Type":"ContainerStarted","Data":"b655ba171005f964ede37e3c7771b5b28cb4d5213c7f65bb3bc3a7bd8e660ec0"}
Sep 10 21:02:42.812900 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:42.812833    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:02:42.830108 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:42.830040    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-767744946f-t79lq" podStartSLOduration=1.039467167 podStartE2EDuration="4.830019649s" totalImagesPullingTime="3.790552482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:42.828272188 +0000 UTC m=+290.748892575" watchObservedRunningTime="2026-09-10 21:02:42.830019649 +0000 UTC m=+290.750640033"
Sep 10 21:02:42.844180 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:42.844127    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl" podStartSLOduration=1.007599645 podStartE2EDuration="4.844110175s" totalImagesPullingTime="3.83651053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:42.843701489 +0000 UTC m=+290.764321900" watchObservedRunningTime="2026-09-10 21:02:42.844110175 +0000 UTC m=+290.764730564"
Sep 10 21:02:52.421169 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:52.421138    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:02:53.819703 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:02:53.819670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f8c8876d7-jxlnl"
Sep 10 21:04:19.560998 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.560919    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"]
Sep 10 21:04:19.568750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.568721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.573209 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.573185    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 21:04:19.573608 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.573587    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 21:04:19.573743 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.573724    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-45qt5\""
Sep 10 21:04:19.574766 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.574743    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:04:19.574879 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.574774    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:04:19.576117 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.576092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwt8\" (UniqueName: \"kubernetes.io/projected/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-kube-api-access-xlwt8\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.576218 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.576139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.576218 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.576198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.576542 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.576521    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"]
Sep 10 21:04:19.676464 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.676427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.676652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.676481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.676652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.676567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwt8\" (UniqueName: \"kubernetes.io/projected/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-kube-api-access-xlwt8\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.676773 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:04:19.676676    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 21:04:19.676773 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:04:19.676758    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert podName:ce6f5883-8f55-4aa0-bb32-0b2ec8adea32 nodeName:}" failed. No retries permitted until 2026-09-10 21:04:20.176737012 +0000 UTC m=+388.097357396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-6zbkc" (UID: "ce6f5883-8f55-4aa0-bb32-0b2ec8adea32") : secret "plugin-serving-cert" not found
Sep 10 21:04:19.677206 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.677183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:19.685936 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:19.685912    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwt8\" (UniqueName: \"kubernetes.io/projected/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-kube-api-access-xlwt8\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:20.180285 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:20.180245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:20.182571 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:20.182551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce6f5883-8f55-4aa0-bb32-0b2ec8adea32-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-6zbkc\" (UID: \"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:20.478332 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:20.478294    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"
Sep 10 21:04:20.750623 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:20.750598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc"]
Sep 10 21:04:20.753063 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:04:20.753034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6f5883_8f55_4aa0_bb32_0b2ec8adea32.slice/crio-911790e290d2939342f2edf3f81f1dd7aa62b61f78ce0e9f70fa67b34893782b WatchSource:0}: Error finding container 911790e290d2939342f2edf3f81f1dd7aa62b61f78ce0e9f70fa67b34893782b: Status 404 returned error can't find the container with id 911790e290d2939342f2edf3f81f1dd7aa62b61f78ce0e9f70fa67b34893782b
Sep 10 21:04:20.754511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:20.754492    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:04:21.173889 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:21.173792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc" event={"ID":"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32","Type":"ContainerStarted","Data":"911790e290d2939342f2edf3f81f1dd7aa62b61f78ce0e9f70fa67b34893782b"}
Sep 10 21:04:26.201443 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:26.201407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc" event={"ID":"ce6f5883-8f55-4aa0-bb32-0b2ec8adea32","Type":"ContainerStarted","Data":"912366a41db245354540be093f9c8276ea9c2e630a487d15c682756debfbd6af"}
Sep 10 21:04:26.217981 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:26.217934    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-6zbkc" podStartSLOduration=2.5964513780000003 podStartE2EDuration="7.217920376s" totalImagesPullingTime="4.621468998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:26.217062637 +0000 UTC m=+394.137683022" watchObservedRunningTime="2026-09-10 21:04:26.217920376 +0000 UTC m=+394.138540762"
Sep 10 21:04:36.075908 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.075875    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868fdd4ff8-x88gh"]
Sep 10 21:04:36.079251 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.079234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.083569 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.083545    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:04:36.083690 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.083640    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 21:04:36.085026 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.085004    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:04:36.085130 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.085077    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:04:36.085130 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.085108    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:04:36.085243 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.085124    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 21:04:36.086223 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.086204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9msmf\""
Sep 10 21:04:36.086402 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.086387    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:04:36.089839 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.089649    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:04:36.090637 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.090616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdd4ff8-x88gh"]
Sep 10 21:04:36.096067 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096043    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbww\" (UniqueName: \"kubernetes.io/projected/7c62f27b-9878-4335-b986-8bcf7f84a9ff-kube-api-access-lqbww\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-oauth-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-oauth-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-trusted-ca-bundle\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.096333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.096222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-service-ca\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197254 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-trusted-ca-bundle\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-service-ca\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbww\" (UniqueName: \"kubernetes.io/projected/7c62f27b-9878-4335-b986-8bcf7f84a9ff-kube-api-access-lqbww\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197375    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197409    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-oauth-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.197654 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.197436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-oauth-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.198167 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.198139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-service-ca\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.198279 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.198149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.198441 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.198414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-trusted-ca-bundle\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.198750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.198729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c62f27b-9878-4335-b986-8bcf7f84a9ff-oauth-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.199899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.199872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-serving-cert\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.199899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.199896    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c62f27b-9878-4335-b986-8bcf7f84a9ff-console-oauth-config\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.205078 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.205053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbww\" (UniqueName: \"kubernetes.io/projected/7c62f27b-9878-4335-b986-8bcf7f84a9ff-kube-api-access-lqbww\") pod \"console-868fdd4ff8-x88gh\" (UID: \"7c62f27b-9878-4335-b986-8bcf7f84a9ff\") " pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.388104 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.388032    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:36.656842 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:36.656800    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868fdd4ff8-x88gh"]
Sep 10 21:04:36.659158 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:04:36.659127    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c62f27b_9878_4335_b986_8bcf7f84a9ff.slice/crio-2e7ff34230282ddd67059bc1d6e06b24104dc1a0fe05b354f8a5d80444a89e4f WatchSource:0}: Error finding container 2e7ff34230282ddd67059bc1d6e06b24104dc1a0fe05b354f8a5d80444a89e4f: Status 404 returned error can't find the container with id 2e7ff34230282ddd67059bc1d6e06b24104dc1a0fe05b354f8a5d80444a89e4f
Sep 10 21:04:37.244644 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:37.244611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdd4ff8-x88gh" event={"ID":"7c62f27b-9878-4335-b986-8bcf7f84a9ff","Type":"ContainerStarted","Data":"ee3ba2f34a5247cb072e9026cad41b96730708628d4a6444adc11b1bc8a295c0"}
Sep 10 21:04:37.245083 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:37.244667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868fdd4ff8-x88gh" event={"ID":"7c62f27b-9878-4335-b986-8bcf7f84a9ff","Type":"ContainerStarted","Data":"2e7ff34230282ddd67059bc1d6e06b24104dc1a0fe05b354f8a5d80444a89e4f"}
Sep 10 21:04:37.264066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:37.264019    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868fdd4ff8-x88gh" podStartSLOduration=1.264004058 podStartE2EDuration="1.264004058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:37.263543058 +0000 UTC m=+405.184163473" watchObservedRunningTime="2026-09-10 21:04:37.264004058 +0000 UTC m=+405.184624445"
Sep 10 21:04:46.388187 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:46.388150    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:46.388556 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:46.388211    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:46.392759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:46.392737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:04:47.285302 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:04:47.285269    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868fdd4ff8-x88gh"
Sep 10 21:05:14.149178 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.149137    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:14.168324 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.168299    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:14.168490 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.168398    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:14.170963 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.170942    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gdskm\""
Sep 10 21:05:14.267005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.266973    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:14.269967 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.269950    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:14.276064 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.276039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:14.283429 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.283407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhmd\" (UniqueName: \"kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd\") pod \"authorino-7bcfd8f9cd-bjqqj\" (UID: \"8e7439a4-3f1a-4846-9b52-8d3253a035e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:14.384010 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.383981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4952\" (UniqueName: \"kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952\") pod \"authorino-59c974bd48-mw8tx\" (UID: \"85deea80-5b21-461e-b63c-3276158416f3\") " pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:14.384172 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.384039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhmd\" (UniqueName: \"kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd\") pod \"authorino-7bcfd8f9cd-bjqqj\" (UID: \"8e7439a4-3f1a-4846-9b52-8d3253a035e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:14.392287 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.392263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhmd\" (UniqueName: \"kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd\") pod \"authorino-7bcfd8f9cd-bjqqj\" (UID: \"8e7439a4-3f1a-4846-9b52-8d3253a035e2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:14.477170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.477143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:14.485322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.485299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4952\" (UniqueName: \"kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952\") pod \"authorino-59c974bd48-mw8tx\" (UID: \"85deea80-5b21-461e-b63c-3276158416f3\") " pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:14.493912 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.493892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4952\" (UniqueName: \"kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952\") pod \"authorino-59c974bd48-mw8tx\" (UID: \"85deea80-5b21-461e-b63c-3276158416f3\") " pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:14.579078 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.579051    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:14.741909 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.741871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:14.743927 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:05:14.743902    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e7439a4_3f1a_4846_9b52_8d3253a035e2.slice/crio-46f194d5ddbd533b5a2950ebebc66abefd234ab092f5d6c6229685c32a6bb0b2 WatchSource:0}: Error finding container 46f194d5ddbd533b5a2950ebebc66abefd234ab092f5d6c6229685c32a6bb0b2: Status 404 returned error can't find the container with id 46f194d5ddbd533b5a2950ebebc66abefd234ab092f5d6c6229685c32a6bb0b2
Sep 10 21:05:14.844993 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:14.844931    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:14.846921 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:05:14.846897    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85deea80_5b21_461e_b63c_3276158416f3.slice/crio-b28cce8be720ab5366ff6264292694754320d04ceb5f6e6ce3d28d5086d14fcf WatchSource:0}: Error finding container b28cce8be720ab5366ff6264292694754320d04ceb5f6e6ce3d28d5086d14fcf: Status 404 returned error can't find the container with id b28cce8be720ab5366ff6264292694754320d04ceb5f6e6ce3d28d5086d14fcf
Sep 10 21:05:15.385096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:15.385056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" event={"ID":"8e7439a4-3f1a-4846-9b52-8d3253a035e2","Type":"ContainerStarted","Data":"46f194d5ddbd533b5a2950ebebc66abefd234ab092f5d6c6229685c32a6bb0b2"}
Sep 10 21:05:15.386218 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:15.386192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mw8tx" event={"ID":"85deea80-5b21-461e-b63c-3276158416f3","Type":"ContainerStarted","Data":"b28cce8be720ab5366ff6264292694754320d04ceb5f6e6ce3d28d5086d14fcf"}
Sep 10 21:05:18.401036 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:18.400993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mw8tx" event={"ID":"85deea80-5b21-461e-b63c-3276158416f3","Type":"ContainerStarted","Data":"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"}
Sep 10 21:05:18.402528 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:18.402498    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" event={"ID":"8e7439a4-3f1a-4846-9b52-8d3253a035e2","Type":"ContainerStarted","Data":"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"}
Sep 10 21:05:18.416083 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:18.416041    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mw8tx" podStartSLOduration=1.6930212180000002 podStartE2EDuration="4.4160275s" totalImagesPullingTime="2.723006282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:18.414423424 +0000 UTC m=+446.335043812" watchObservedRunningTime="2026-09-10 21:05:18.4160275 +0000 UTC m=+446.336647886"
Sep 10 21:05:18.427745 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:18.427706    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" podStartSLOduration=1.611655661 podStartE2EDuration="4.42769354s" totalImagesPullingTime="2.816037879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:18.427058843 +0000 UTC m=+446.347679230" watchObservedRunningTime="2026-09-10 21:05:18.42769354 +0000 UTC m=+446.348313925"
Sep 10 21:05:18.450425 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:18.450397    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:20.409852 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:20.409795    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" podUID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" containerName="authorino" containerID="cri-o://7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645" gracePeriod=30
Sep 10 21:05:20.643672 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:20.643649    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:20.742293 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:20.742264    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlhmd\" (UniqueName: \"kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd\") pod \"8e7439a4-3f1a-4846-9b52-8d3253a035e2\" (UID: \"8e7439a4-3f1a-4846-9b52-8d3253a035e2\") "
Sep 10 21:05:20.744353 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:20.744326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd" (OuterVolumeSpecName: "kube-api-access-qlhmd") pod "8e7439a4-3f1a-4846-9b52-8d3253a035e2" (UID: "8e7439a4-3f1a-4846-9b52-8d3253a035e2"). InnerVolumeSpecName "kube-api-access-qlhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:20.843409 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:20.843379    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhmd\" (UniqueName: \"kubernetes.io/projected/8e7439a4-3f1a-4846-9b52-8d3253a035e2-kube-api-access-qlhmd\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:05:21.415158 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.415124    2562 generic.go:356] "Generic (PLEG): container finished" podID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" containerID="7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645" exitCode=0
Sep 10 21:05:21.415610 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.415162    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj"
Sep 10 21:05:21.415610 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.415217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" event={"ID":"8e7439a4-3f1a-4846-9b52-8d3253a035e2","Type":"ContainerDied","Data":"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"}
Sep 10 21:05:21.415610 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.415255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-bjqqj" event={"ID":"8e7439a4-3f1a-4846-9b52-8d3253a035e2","Type":"ContainerDied","Data":"46f194d5ddbd533b5a2950ebebc66abefd234ab092f5d6c6229685c32a6bb0b2"}
Sep 10 21:05:21.415610 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.415271    2562 scope.go:117] "RemoveContainer" containerID="7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"
Sep 10 21:05:21.423558 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.423540    2562 scope.go:117] "RemoveContainer" containerID="7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"
Sep 10 21:05:21.423847 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:05:21.423826    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645\": container with ID starting with 7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645 not found: ID does not exist" containerID="7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"
Sep 10 21:05:21.423940 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.423851    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645"} err="failed to get container status \"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645\": rpc error: code = NotFound desc = could not find container \"7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645\": container with ID starting with 7f5e65b0b5125efd8f4cec5732abfccd591ad145eb8f43daa2cee12082a8e645 not found: ID does not exist"
Sep 10 21:05:21.435401 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.435379    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:21.438665 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:21.438643    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-bjqqj"]
Sep 10 21:05:22.617857 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:22.617792    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" path="/var/lib/kubelet/pods/8e7439a4-3f1a-4846-9b52-8d3253a035e2/volumes"
Sep 10 21:05:41.916010 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.915980    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-4gkbs"]
Sep 10 21:05:41.916384 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.916243    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" containerName="authorino"
Sep 10 21:05:41.916384 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.916254    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" containerName="authorino"
Sep 10 21:05:41.916384 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.916324    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7439a4-3f1a-4846-9b52-8d3253a035e2" containerName="authorino"
Sep 10 21:05:41.920459 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.920434    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:41.926025 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.926001    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-4gkbs"]
Sep 10 21:05:41.968701 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.968676    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4gkbs"]
Sep 10 21:05:41.968910 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:05:41.968892    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cjvmt], unattached volumes=[], failed to process volumes=[kube-api-access-cjvmt]: context canceled" pod="kuadrant-system/authorino-557f876f69-4gkbs" podUID="698b7b1a-e07f-4b8b-b154-7ba7746780f7"
Sep 10 21:05:41.990959 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.990908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvmt\" (UniqueName: \"kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt\") pod \"authorino-557f876f69-4gkbs\" (UID: \"698b7b1a-e07f-4b8b-b154-7ba7746780f7\") " pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:41.992477 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.992458    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fm79q"]
Sep 10 21:05:41.995396 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.995377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:41.997840 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:41.997825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:05:42.002421 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.002397    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fm79q"]
Sep 10 21:05:42.091351 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.091319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.091485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.091354    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jvx\" (UniqueName: \"kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.091485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.091403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvmt\" (UniqueName: \"kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt\") pod \"authorino-557f876f69-4gkbs\" (UID: \"698b7b1a-e07f-4b8b-b154-7ba7746780f7\") " pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:42.099600 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.099573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvmt\" (UniqueName: \"kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt\") pod \"authorino-557f876f69-4gkbs\" (UID: \"698b7b1a-e07f-4b8b-b154-7ba7746780f7\") " pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:42.151792 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.151766    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fm79q"]
Sep 10 21:05:42.151996 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:05:42.151977    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-28jvx tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-fm79q" podUID="bdc64185-2269-4f12-81d8-7ba2c03d92f6"
Sep 10 21:05:42.176621 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.176564    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:05:42.179575 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.179558    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.186180 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.186150    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:05:42.192080 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.192058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfqsr\" (UniqueName: \"kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.192165 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.192094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.192165 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.192120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jvx\" (UniqueName: \"kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.192165 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.192149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.194491 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.194468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.201600 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.201580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jvx\" (UniqueName: \"kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx\") pod \"authorino-6d4dfc649f-fm79q\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") " pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.292488 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.292455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.292654 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.292534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfqsr\" (UniqueName: \"kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.294931 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.294905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.302326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.302296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfqsr\" (UniqueName: \"kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr\") pod \"authorino-758d5f7968-gf9f2\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") " pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.489951 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.489923    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:05:42.502350 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.502328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.502458 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.502328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:42.509256 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.509236    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:42.512089 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.512070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:42.595039 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.595009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjvmt\" (UniqueName: \"kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt\") pod \"698b7b1a-e07f-4b8b-b154-7ba7746780f7\" (UID: \"698b7b1a-e07f-4b8b-b154-7ba7746780f7\") "
Sep 10 21:05:42.595194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.595068    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jvx\" (UniqueName: \"kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx\") pod \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") "
Sep 10 21:05:42.595194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.595108    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert\") pod \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\" (UID: \"bdc64185-2269-4f12-81d8-7ba2c03d92f6\") "
Sep 10 21:05:42.597084 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.597049    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx" (OuterVolumeSpecName: "kube-api-access-28jvx") pod "bdc64185-2269-4f12-81d8-7ba2c03d92f6" (UID: "bdc64185-2269-4f12-81d8-7ba2c03d92f6"). InnerVolumeSpecName "kube-api-access-28jvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:42.597189 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.597123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "bdc64185-2269-4f12-81d8-7ba2c03d92f6" (UID: "bdc64185-2269-4f12-81d8-7ba2c03d92f6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:42.597189 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.597140    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt" (OuterVolumeSpecName: "kube-api-access-cjvmt") pod "698b7b1a-e07f-4b8b-b154-7ba7746780f7" (UID: "698b7b1a-e07f-4b8b-b154-7ba7746780f7"). InnerVolumeSpecName "kube-api-access-cjvmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:42.696496 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.696455    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bdc64185-2269-4f12-81d8-7ba2c03d92f6-tls-cert\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:05:42.696496 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.696493    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvmt\" (UniqueName: \"kubernetes.io/projected/698b7b1a-e07f-4b8b-b154-7ba7746780f7-kube-api-access-cjvmt\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:05:42.696724 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.696507    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jvx\" (UniqueName: \"kubernetes.io/projected/bdc64185-2269-4f12-81d8-7ba2c03d92f6-kube-api-access-28jvx\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:05:42.760122 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:42.760053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:05:42.763777 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:05:42.763752    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffe9ebe1_fa88_4777_8393_2d59f9994434.slice/crio-27c3b0aad0dcb13513b060bab6cd34cdb2ca59f316011b7915208e9d8571ba7e WatchSource:0}: Error finding container 27c3b0aad0dcb13513b060bab6cd34cdb2ca59f316011b7915208e9d8571ba7e: Status 404 returned error can't find the container with id 27c3b0aad0dcb13513b060bab6cd34cdb2ca59f316011b7915208e9d8571ba7e
Sep 10 21:05:43.507202 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.507177    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-4gkbs"
Sep 10 21:05:43.507582 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.507214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fm79q"
Sep 10 21:05:43.507582 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.507182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d5f7968-gf9f2" event={"ID":"ffe9ebe1-fa88-4777-8393-2d59f9994434","Type":"ContainerStarted","Data":"27c3b0aad0dcb13513b060bab6cd34cdb2ca59f316011b7915208e9d8571ba7e"}
Sep 10 21:05:43.532579 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.532552    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fm79q"]
Sep 10 21:05:43.536201 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.536178    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fm79q"]
Sep 10 21:05:43.558652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.558603    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-4gkbs"]
Sep 10 21:05:43.562602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:43.562577    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-4gkbs"]
Sep 10 21:05:44.512487 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.512449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d5f7968-gf9f2" event={"ID":"ffe9ebe1-fa88-4777-8393-2d59f9994434","Type":"ContainerStarted","Data":"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"}
Sep 10 21:05:44.529396 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.529346    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-758d5f7968-gf9f2" podStartSLOduration=1.8307511170000001 podStartE2EDuration="2.529329343s" totalImagesPullingTime="698.578226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:44.52794762 +0000 UTC m=+472.448568006" watchObservedRunningTime="2026-09-10 21:05:44.529329343 +0000 UTC m=+472.449949729"
Sep 10 21:05:44.551091 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.551059    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:44.551274 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.551250    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mw8tx" podUID="85deea80-5b21-461e-b63c-3276158416f3" containerName="authorino" containerID="cri-o://2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa" gracePeriod=30
Sep 10 21:05:44.620923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.620887    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="698b7b1a-e07f-4b8b-b154-7ba7746780f7" path="/var/lib/kubelet/pods/698b7b1a-e07f-4b8b-b154-7ba7746780f7/volumes"
Sep 10 21:05:44.621199 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.621183    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc64185-2269-4f12-81d8-7ba2c03d92f6" path="/var/lib/kubelet/pods/bdc64185-2269-4f12-81d8-7ba2c03d92f6/volumes"
Sep 10 21:05:44.799503 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.799480    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:44.809886 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.809860    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4952\" (UniqueName: \"kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952\") pod \"85deea80-5b21-461e-b63c-3276158416f3\" (UID: \"85deea80-5b21-461e-b63c-3276158416f3\") "
Sep 10 21:05:44.811734 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.811713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952" (OuterVolumeSpecName: "kube-api-access-b4952") pod "85deea80-5b21-461e-b63c-3276158416f3" (UID: "85deea80-5b21-461e-b63c-3276158416f3"). InnerVolumeSpecName "kube-api-access-b4952". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:44.911016 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:44.910989    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4952\" (UniqueName: \"kubernetes.io/projected/85deea80-5b21-461e-b63c-3276158416f3-kube-api-access-b4952\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:05:45.518151 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.518119    2562 generic.go:356] "Generic (PLEG): container finished" podID="85deea80-5b21-461e-b63c-3276158416f3" containerID="2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa" exitCode=0
Sep 10 21:05:45.518548 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.518165    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mw8tx"
Sep 10 21:05:45.518548 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.518159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mw8tx" event={"ID":"85deea80-5b21-461e-b63c-3276158416f3","Type":"ContainerDied","Data":"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"}
Sep 10 21:05:45.518548 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.518278    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mw8tx" event={"ID":"85deea80-5b21-461e-b63c-3276158416f3","Type":"ContainerDied","Data":"b28cce8be720ab5366ff6264292694754320d04ceb5f6e6ce3d28d5086d14fcf"}
Sep 10 21:05:45.518548 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.518307    2562 scope.go:117] "RemoveContainer" containerID="2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"
Sep 10 21:05:45.526465 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.526449    2562 scope.go:117] "RemoveContainer" containerID="2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"
Sep 10 21:05:45.526729 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:05:45.526704    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa\": container with ID starting with 2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa not found: ID does not exist" containerID="2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"
Sep 10 21:05:45.526835 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.526735    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa"} err="failed to get container status \"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa\": rpc error: code = NotFound desc = could not find container \"2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa\": container with ID starting with 2bb80432d6775a06a19f34a5e9c25c5da7513afeb6b382efda5259286add0daa not found: ID does not exist"
Sep 10 21:05:45.541505 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.541484    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:45.543852 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:45.543832    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mw8tx"]
Sep 10 21:05:46.619741 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:05:46.619709    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85deea80-5b21-461e-b63c-3276158416f3" path="/var/lib/kubelet/pods/85deea80-5b21-461e-b63c-3276158416f3/volumes"
Sep 10 21:06:25.277009 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.276971    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"]
Sep 10 21:06:25.277453 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.277241    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85deea80-5b21-461e-b63c-3276158416f3" containerName="authorino"
Sep 10 21:06:25.277453 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.277253    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="85deea80-5b21-461e-b63c-3276158416f3" containerName="authorino"
Sep 10 21:06:25.277453 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.277318    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="85deea80-5b21-461e-b63c-3276158416f3" containerName="authorino"
Sep 10 21:06:25.280159 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.280141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.282817 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.282780    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 21:06:25.282934 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.282917    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kqq99\""
Sep 10 21:06:25.289128 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.289102    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"]
Sep 10 21:06:25.414179 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.414138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqvq\" (UniqueName: \"kubernetes.io/projected/3a68c30c-63f3-45c1-b656-976e1710d528-kube-api-access-rnqvq\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.414339 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.414189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a68c30c-63f3-45c1-b656-976e1710d528-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.514875 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.514836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqvq\" (UniqueName: \"kubernetes.io/projected/3a68c30c-63f3-45c1-b656-976e1710d528-kube-api-access-rnqvq\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.515051 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.514887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a68c30c-63f3-45c1-b656-976e1710d528-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.515609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.515584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a68c30c-63f3-45c1-b656-976e1710d528-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.523007 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.522980    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqvq\" (UniqueName: \"kubernetes.io/projected/3a68c30c-63f3-45c1-b656-976e1710d528-kube-api-access-rnqvq\") pod \"payload-processing-5b54b7fbb7-6fs7j\" (UID: \"3a68c30c-63f3-45c1-b656-976e1710d528\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.590909 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.590834    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:25.863048 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:25.863023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"]
Sep 10 21:06:25.865338 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:06:25.865310    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a68c30c_63f3_45c1_b656_976e1710d528.slice/crio-073859e6750815d9c0e37b38e5b3e7f0d3be63ea23bdaddc28d567df6fcdb739 WatchSource:0}: Error finding container 073859e6750815d9c0e37b38e5b3e7f0d3be63ea23bdaddc28d567df6fcdb739: Status 404 returned error can't find the container with id 073859e6750815d9c0e37b38e5b3e7f0d3be63ea23bdaddc28d567df6fcdb739
Sep 10 21:06:26.681312 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:26.681275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j" event={"ID":"3a68c30c-63f3-45c1-b656-976e1710d528","Type":"ContainerStarted","Data":"073859e6750815d9c0e37b38e5b3e7f0d3be63ea23bdaddc28d567df6fcdb739"}
Sep 10 21:06:28.689435 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:28.689399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j" event={"ID":"3a68c30c-63f3-45c1-b656-976e1710d528","Type":"ContainerStarted","Data":"b17db12f0237c2c5394c693ba8916ef5a8cac377e40aee85698bea50656e3960"}
Sep 10 21:06:28.689791 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:28.689539    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:28.705285 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:28.705242    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j" podStartSLOduration=1.88642607 podStartE2EDuration="3.705224477s" totalImagesPullingTime="1.818798407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:28.704833131 +0000 UTC m=+516.625453544" watchObservedRunningTime="2026-09-10 21:06:28.705224477 +0000 UTC m=+516.625844865"
Sep 10 21:06:39.695794 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:39.695764    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-6fs7j"
Sep 10 21:06:58.855338 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.855248    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"]
Sep 10 21:06:58.858597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.858575    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.862902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.862867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mtwrf\""
Sep 10 21:06:58.863041 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.862963    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 21:06:58.863041 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.863010    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:06:58.863160 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.863097    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:06:58.868293 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.868261    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"]
Sep 10 21:06:58.880081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.880211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880092    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.880211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.880211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.880211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zsp\" (UniqueName: \"kubernetes.io/projected/86168c85-b00f-4b80-adc8-839c0239a99a-kube-api-access-26zsp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.880346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.880214    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86168c85-b00f-4b80-adc8-839c0239a99a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zsp\" (UniqueName: \"kubernetes.io/projected/86168c85-b00f-4b80-adc8-839c0239a99a-kube-api-access-26zsp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981227    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86168c85-b00f-4b80-adc8-839c0239a99a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.981750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.981734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.983591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.983574    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86168c85-b00f-4b80-adc8-839c0239a99a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.983736 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.983718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86168c85-b00f-4b80-adc8-839c0239a99a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:58.988486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:58.988469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zsp\" (UniqueName: \"kubernetes.io/projected/86168c85-b00f-4b80-adc8-839c0239a99a-kube-api-access-26zsp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lmc2c\" (UID: \"86168c85-b00f-4b80-adc8-839c0239a99a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:59.172131 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:59.172044    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:06:59.440140 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:59.439974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"]
Sep 10 21:06:59.442287 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:06:59.442260    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86168c85_b00f_4b80_adc8_839c0239a99a.slice/crio-770520ab6579754efde4746cbc5ea32a069e9b0765d26db10bf96c07eddee2f3 WatchSource:0}: Error finding container 770520ab6579754efde4746cbc5ea32a069e9b0765d26db10bf96c07eddee2f3: Status 404 returned error can't find the container with id 770520ab6579754efde4746cbc5ea32a069e9b0765d26db10bf96c07eddee2f3
Sep 10 21:06:59.821584 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:06:59.821489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c" event={"ID":"86168c85-b00f-4b80-adc8-839c0239a99a","Type":"ContainerStarted","Data":"770520ab6579754efde4746cbc5ea32a069e9b0765d26db10bf96c07eddee2f3"}
Sep 10 21:07:00.826636 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:00.826598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c" event={"ID":"86168c85-b00f-4b80-adc8-839c0239a99a","Type":"ContainerStarted","Data":"6954a66ff5ef20f186b2576de02b78d202ac6f233040e9c4436672934f946175"}
Sep 10 21:07:00.843311 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:00.843218    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c" podStartSLOduration=1.6946560910000001 podStartE2EDuration="2.843203937s" totalImagesPullingTime="1.148547846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:00.842640407 +0000 UTC m=+548.763260804" watchObservedRunningTime="2026-09-10 21:07:00.843203937 +0000 UTC m=+548.763824324"
Sep 10 21:07:09.172425 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:09.172387    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:07:09.172923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:09.172450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:07:09.184739 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:09.184715    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:07:09.872974 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:09.872945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lmc2c"
Sep 10 21:07:17.258257 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.258222    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"]
Sep 10 21:07:17.261550 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.261529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.264170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.264149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 21:07:17.270428 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.270400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"]
Sep 10 21:07:17.313630 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313591    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.313800 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.313800 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313737    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvr5q\" (UniqueName: \"kubernetes.io/projected/83dc4926-909e-4df2-9489-5803dd297c1d-kube-api-access-fvr5q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.313800 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.313800 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.313995 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.313856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83dc4926-909e-4df2-9489-5803dd297c1d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415049 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvr5q\" (UniqueName: \"kubernetes.io/projected/83dc4926-909e-4df2-9489-5803dd297c1d-kube-api-access-fvr5q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415123    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83dc4926-909e-4df2-9489-5803dd297c1d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415147    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415526 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415568 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.415568 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.415551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.417456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.417432    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83dc4926-909e-4df2-9489-5803dd297c1d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.417604 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.417582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83dc4926-909e-4df2-9489-5803dd297c1d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.423035 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.423014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvr5q\" (UniqueName: \"kubernetes.io/projected/83dc4926-909e-4df2-9489-5803dd297c1d-kube-api-access-fvr5q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vbltj\" (UID: \"83dc4926-909e-4df2-9489-5803dd297c1d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.571883 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.571792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:17.839834 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.839794    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"]
Sep 10 21:07:17.841959 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:07:17.841927    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dc4926_909e_4df2_9489_5803dd297c1d.slice/crio-5fb17d756dc9b43312041e2951e310ee3ecb92416d421b450a4f71affe6f934e WatchSource:0}: Error finding container 5fb17d756dc9b43312041e2951e310ee3ecb92416d421b450a4f71affe6f934e: Status 404 returned error can't find the container with id 5fb17d756dc9b43312041e2951e310ee3ecb92416d421b450a4f71affe6f934e
Sep 10 21:07:17.894020 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:17.893991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj" event={"ID":"83dc4926-909e-4df2-9489-5803dd297c1d","Type":"ContainerStarted","Data":"5fb17d756dc9b43312041e2951e310ee3ecb92416d421b450a4f71affe6f934e"}
Sep 10 21:07:18.899605 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:18.899566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj" event={"ID":"83dc4926-909e-4df2-9489-5803dd297c1d","Type":"ContainerStarted","Data":"643268d35d91c26413e2a22197eccfb618666c3042331f27dbcc12e1cde80e83"}
Sep 10 21:07:18.916489 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:18.916438    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj" podStartSLOduration=1.6896406370000001 podStartE2EDuration="1.916419911s" totalImagesPullingTime="226.779274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:18.916280263 +0000 UTC m=+566.836900649" watchObservedRunningTime="2026-09-10 21:07:18.916419911 +0000 UTC m=+566.837040300"
Sep 10 21:07:27.572708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:27.572665    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:27.573146 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:27.572738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:27.585555 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:27.585528    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:27.952301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:27.952275    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vbltj"
Sep 10 21:07:57.855952 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:57.855916    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"]
Sep 10 21:07:57.858936 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:57.858919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:57.861570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:57.861543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 21:07:57.868998 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:57.868977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"]
Sep 10 21:07:58.042237 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvq8\" (UniqueName: \"kubernetes.io/projected/7fd01427-bb09-430d-a4ed-055927b653d1-kube-api-access-jtvq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.042406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.042406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd01427-bb09-430d-a4ed-055927b653d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.042406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.042406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.042556 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.042412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvq8\" (UniqueName: \"kubernetes.io/projected/7fd01427-bb09-430d-a4ed-055927b653d1-kube-api-access-jtvq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143500    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd01427-bb09-430d-a4ed-055927b653d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143984 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143984 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.143984 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.143926    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.145723 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.145701    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fd01427-bb09-430d-a4ed-055927b653d1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.146012 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.145995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fd01427-bb09-430d-a4ed-055927b653d1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.151409 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.151384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvq8\" (UniqueName: \"kubernetes.io/projected/7fd01427-bb09-430d-a4ed-055927b653d1-kube-api-access-jtvq8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk\" (UID: \"7fd01427-bb09-430d-a4ed-055927b653d1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.167997 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.167963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:07:58.436820 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:58.436778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"]
Sep 10 21:07:58.439228 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:07:58.439202    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fd01427_bb09_430d_a4ed_055927b653d1.slice/crio-81b723e8c7389b4c9ab419ea1947221f8cb74ae95b29971bb8b30bf7b5334c01 WatchSource:0}: Error finding container 81b723e8c7389b4c9ab419ea1947221f8cb74ae95b29971bb8b30bf7b5334c01: Status 404 returned error can't find the container with id 81b723e8c7389b4c9ab419ea1947221f8cb74ae95b29971bb8b30bf7b5334c01
Sep 10 21:07:59.062696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:59.062660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk" event={"ID":"7fd01427-bb09-430d-a4ed-055927b653d1","Type":"ContainerStarted","Data":"88e55f98f0462d21eace0fb08890e246bb062ef77182d5e8fca8e603a408f6be"}
Sep 10 21:07:59.063071 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:59.062704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk" event={"ID":"7fd01427-bb09-430d-a4ed-055927b653d1","Type":"ContainerStarted","Data":"81b723e8c7389b4c9ab419ea1947221f8cb74ae95b29971bb8b30bf7b5334c01"}
Sep 10 21:07:59.081564 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:07:59.081519    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk" podStartSLOduration=1.8778573710000002 podStartE2EDuration="2.081502301s" totalImagesPullingTime="203.64493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:59.080104143 +0000 UTC m=+607.000724540" watchObservedRunningTime="2026-09-10 21:07:59.081502301 +0000 UTC m=+607.002122688"
Sep 10 21:08:08.168330 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:08.168300    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:08:08.168755 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:08.168485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:08:08.181270 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:08.181231    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:08:09.117315 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:09.117288    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk"
Sep 10 21:08:58.604072 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.604037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b58cd5757-8dmv6"]
Sep 10 21:08:58.607243 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.607226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.618851 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.618827    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b58cd5757-8dmv6"]
Sep 10 21:08:58.710882 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.710853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpwc\" (UniqueName: \"kubernetes.io/projected/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-kube-api-access-vrpwc\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.711005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.710894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-tls-cert\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.811235 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.811209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpwc\" (UniqueName: \"kubernetes.io/projected/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-kube-api-access-vrpwc\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.811349 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.811240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-tls-cert\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.813499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.813478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-tls-cert\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.818581 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.818560    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpwc\" (UniqueName: \"kubernetes.io/projected/7b104a17-3c4f-4bfc-a5fc-cc8b8989b886-kube-api-access-vrpwc\") pod \"authorino-7b58cd5757-8dmv6\" (UID: \"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886\") " pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:58.917319 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:58.917231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b58cd5757-8dmv6"
Sep 10 21:08:59.182615 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:59.182537    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b58cd5757-8dmv6"]
Sep 10 21:08:59.185737 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:08:59.185709    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b104a17_3c4f_4bfc_a5fc_cc8b8989b886.slice/crio-c1e098018b4694b4d7e17b93ebe939180d83605b7e729785cdba1ade4b3f3692 WatchSource:0}: Error finding container c1e098018b4694b4d7e17b93ebe939180d83605b7e729785cdba1ade4b3f3692: Status 404 returned error can't find the container with id c1e098018b4694b4d7e17b93ebe939180d83605b7e729785cdba1ade4b3f3692
Sep 10 21:08:59.318232 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:08:59.318197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b58cd5757-8dmv6" event={"ID":"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886","Type":"ContainerStarted","Data":"c1e098018b4694b4d7e17b93ebe939180d83605b7e729785cdba1ade4b3f3692"}
Sep 10 21:09:00.324999 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.324960    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b58cd5757-8dmv6" event={"ID":"7b104a17-3c4f-4bfc-a5fc-cc8b8989b886","Type":"ContainerStarted","Data":"68a33246c6946c08285ec09794e1306a78f8f3fd27ca6391b8e7a5f2137b0d90"}
Sep 10 21:09:00.342208 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.342158    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b58cd5757-8dmv6" podStartSLOduration=1.775655784 podStartE2EDuration="2.342139973s" totalImagesPullingTime="566.484189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:00.341306944 +0000 UTC m=+668.261927332" watchObservedRunningTime="2026-09-10 21:09:00.342139973 +0000 UTC m=+668.262760359"
Sep 10 21:09:00.370015 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.369982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:09:00.370737 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.370699    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-758d5f7968-gf9f2" podUID="ffe9ebe1-fa88-4777-8393-2d59f9994434" containerName="authorino" containerID="cri-o://b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c" gracePeriod=30
Sep 10 21:09:00.617043 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.617020    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:09:00.724531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.724493    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert\") pod \"ffe9ebe1-fa88-4777-8393-2d59f9994434\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") "
Sep 10 21:09:00.724693 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.724633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfqsr\" (UniqueName: \"kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr\") pod \"ffe9ebe1-fa88-4777-8393-2d59f9994434\" (UID: \"ffe9ebe1-fa88-4777-8393-2d59f9994434\") "
Sep 10 21:09:00.726614 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.726580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr" (OuterVolumeSpecName: "kube-api-access-gfqsr") pod "ffe9ebe1-fa88-4777-8393-2d59f9994434" (UID: "ffe9ebe1-fa88-4777-8393-2d59f9994434"). InnerVolumeSpecName "kube-api-access-gfqsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:00.735206 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.735176    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ffe9ebe1-fa88-4777-8393-2d59f9994434" (UID: "ffe9ebe1-fa88-4777-8393-2d59f9994434"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:09:00.825915 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.825878    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ffe9ebe1-fa88-4777-8393-2d59f9994434-tls-cert\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:09:00.826034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:00.825919    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfqsr\" (UniqueName: \"kubernetes.io/projected/ffe9ebe1-fa88-4777-8393-2d59f9994434-kube-api-access-gfqsr\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:09:01.330357 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.330318    2562 generic.go:356] "Generic (PLEG): container finished" podID="ffe9ebe1-fa88-4777-8393-2d59f9994434" containerID="b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c" exitCode=0
Sep 10 21:09:01.330357 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.330355    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-758d5f7968-gf9f2"
Sep 10 21:09:01.330965 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.330385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d5f7968-gf9f2" event={"ID":"ffe9ebe1-fa88-4777-8393-2d59f9994434","Type":"ContainerDied","Data":"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"}
Sep 10 21:09:01.330965 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.330425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-758d5f7968-gf9f2" event={"ID":"ffe9ebe1-fa88-4777-8393-2d59f9994434","Type":"ContainerDied","Data":"27c3b0aad0dcb13513b060bab6cd34cdb2ca59f316011b7915208e9d8571ba7e"}
Sep 10 21:09:01.330965 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.330441    2562 scope.go:117] "RemoveContainer" containerID="b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"
Sep 10 21:09:01.340183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.340165    2562 scope.go:117] "RemoveContainer" containerID="b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"
Sep 10 21:09:01.340461 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:01.340442    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c\": container with ID starting with b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c not found: ID does not exist" containerID="b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"
Sep 10 21:09:01.340533 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.340474    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c"} err="failed to get container status \"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c\": rpc error: code = NotFound desc = could not find container \"b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c\": container with ID starting with b162e2e03e8c087f9fdfcd7ddc25b112d5f14d3067000f03e9cc721d1a513e2c not found: ID does not exist"
Sep 10 21:09:01.353895 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.353864    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:09:01.357721 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:01.357693    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-758d5f7968-gf9f2"]
Sep 10 21:09:02.617267 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:02.617230    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffe9ebe1-fa88-4777-8393-2d59f9994434" path="/var/lib/kubelet/pods/ffe9ebe1-fa88-4777-8393-2d59f9994434/volumes"
Sep 10 21:09:18.951176 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.951144    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7746-30d6f4"]
Sep 10 21:09:18.951518 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.951417    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffe9ebe1-fa88-4777-8393-2d59f9994434" containerName="authorino"
Sep 10 21:09:18.951518 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.951429    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffe9ebe1-fa88-4777-8393-2d59f9994434" containerName="authorino"
Sep 10 21:09:18.951518 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.951483    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffe9ebe1-fa88-4777-8393-2d59f9994434" containerName="authorino"
Sep 10 21:09:18.953988 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.953972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:18.956439 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.956417    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8md8h\""
Sep 10 21:09:18.960393 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:18.960373    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-30d6f4"]
Sep 10 21:09:19.081461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:19.081416    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw6j\" (UniqueName: \"kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j\") pod \"test-curl-7746-30d6f4\" (UID: \"56a8b17c-a060-49b5-9efc-bc43c14a515a\") " pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:19.182428 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:19.182390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw6j\" (UniqueName: \"kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j\") pod \"test-curl-7746-30d6f4\" (UID: \"56a8b17c-a060-49b5-9efc-bc43c14a515a\") " pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:19.190688 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:19.190661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw6j\" (UniqueName: \"kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j\") pod \"test-curl-7746-30d6f4\" (UID: \"56a8b17c-a060-49b5-9efc-bc43c14a515a\") " pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:19.263761 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:19.263675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:19.549280 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:19.549199    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-30d6f4"]
Sep 10 21:09:19.552028 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:19.551998    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a8b17c_a060_49b5_9efc_bc43c14a515a.slice/crio-582af19e510e20f9735129e784ae00b0ae79f6e256baf51beba4191e5c2d9666 WatchSource:0}: Error finding container 582af19e510e20f9735129e784ae00b0ae79f6e256baf51beba4191e5c2d9666: Status 404 returned error can't find the container with id 582af19e510e20f9735129e784ae00b0ae79f6e256baf51beba4191e5c2d9666
Sep 10 21:09:20.410679 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:20.410635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-30d6f4" event={"ID":"56a8b17c-a060-49b5-9efc-bc43c14a515a","Type":"ContainerStarted","Data":"582af19e510e20f9735129e784ae00b0ae79f6e256baf51beba4191e5c2d9666"}
Sep 10 21:09:22.313021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.312986    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:09:22.315852 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.315835    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.318485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.318469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a41a24\""
Sep 10 21:09:22.318485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.318476    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a41a24-dockercfg-5f2xb\""
Sep 10 21:09:22.324445 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.324421    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:09:22.408734 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.408685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.408941 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.408866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsxmx\" (UniqueName: \"kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.410585 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.410560    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:09:22.414114 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.414099    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.419899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.419873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-30d6f4" event={"ID":"56a8b17c-a060-49b5-9efc-bc43c14a515a","Type":"ContainerStarted","Data":"96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05"}
Sep 10 21:09:22.424416 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.424395    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:09:22.445749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.445697    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7746-30d6f4" podStartSLOduration=2.530654084 podStartE2EDuration="4.445682647s" totalImagesPullingTime="1.915028563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:22.445406524 +0000 UTC m=+690.366026912" watchObservedRunningTime="2026-09-10 21:09:22.445682647 +0000 UTC m=+690.366303032"
Sep 10 21:09:22.509316 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.509272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.509499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.509333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsxmx\" (UniqueName: \"kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.509963 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.509942    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.517154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.517133    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsxmx\" (UniqueName: \"kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx\") pod \"payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.609725 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.609689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc4f\" (UniqueName: \"kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.609899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.609770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.626426 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.626403    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:22.710510 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.710419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc4f\" (UniqueName: \"kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.710510 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.710473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.711428 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.711404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.722116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.722093    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc4f\" (UniqueName: \"kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f\") pod \"payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.725899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.725875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:22.895849 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.895818    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:09:22.898094 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:22.898072    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935d9dc6_0a54_4234_8249_f302f6c280f1.slice/crio-08676b769e001566f9a7ec1a0879295366bb8670316e10fbaf784044ec6562b2 WatchSource:0}: Error finding container 08676b769e001566f9a7ec1a0879295366bb8670316e10fbaf784044ec6562b2: Status 404 returned error can't find the container with id 08676b769e001566f9a7ec1a0879295366bb8670316e10fbaf784044ec6562b2
Sep 10 21:09:22.996090 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.996004    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7746-30d6f4"]
Sep 10 21:09:22.997910 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.997842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:09:22.999407 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:22.999301    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7746-30d6f4"]
Sep 10 21:09:23.000516 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:23.000483    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9455174c_49e5_4f42_89fc_d2245cbd6e59.slice/crio-3a3568230ba8a62963a233cb56bdf7453464a2b2e04e1ee08395c4cf767407a2 WatchSource:0}: Error finding container 3a3568230ba8a62963a233cb56bdf7453464a2b2e04e1ee08395c4cf767407a2: Status 404 returned error can't find the container with id 3a3568230ba8a62963a233cb56bdf7453464a2b2e04e1ee08395c4cf767407a2
Sep 10 21:09:23.114918 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.114884    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7746-4f0df7"]
Sep 10 21:09:23.115334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.115318    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" containerName="test-curl-7746-30d6f4"
Sep 10 21:09:23.115381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.115337    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" containerName="test-curl-7746-30d6f4"
Sep 10 21:09:23.115429 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.115417    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" containerName="test-curl-7746-30d6f4"
Sep 10 21:09:23.118535 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.118515    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:23.123257 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.123232    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-4f0df7"]
Sep 10 21:09:23.214331 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.214298    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkf8\" (UniqueName: \"kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8\") pod \"test-curl-7746-4f0df7\" (UID: \"b2a58b9f-914b-4807-9f2e-1ca514787528\") " pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:23.314632 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.314595    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkf8\" (UniqueName: \"kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8\") pod \"test-curl-7746-4f0df7\" (UID: \"b2a58b9f-914b-4807-9f2e-1ca514787528\") " pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:23.322046 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.322012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkf8\" (UniqueName: \"kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8\") pod \"test-curl-7746-4f0df7\" (UID: \"b2a58b9f-914b-4807-9f2e-1ca514787528\") " pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:23.426467 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.426429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" event={"ID":"9455174c-49e5-4f42-89fc-d2245cbd6e59","Type":"ContainerStarted","Data":"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"}
Sep 10 21:09:23.426656 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.426533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:23.426656 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.426559    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" event={"ID":"9455174c-49e5-4f42-89fc-d2245cbd6e59","Type":"ContainerStarted","Data":"3a3568230ba8a62963a233cb56bdf7453464a2b2e04e1ee08395c4cf767407a2"}
Sep 10 21:09:23.428107 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.428078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" event={"ID":"935d9dc6-0a54-4234-8249-f302f6c280f1","Type":"ContainerStarted","Data":"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"}
Sep 10 21:09:23.428213 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.428131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" event={"ID":"935d9dc6-0a54-4234-8249-f302f6c280f1","Type":"ContainerStarted","Data":"08676b769e001566f9a7ec1a0879295366bb8670316e10fbaf784044ec6562b2"}
Sep 10 21:09:23.428327 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:23.428310    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7746-30d6f4" container="test-curl-7746-30d6f4"
Sep 10 21:09:23.429709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.429692    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:23.445571 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.445519    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" podStartSLOduration=1.445500306 podStartE2EDuration="1.445500306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:23.443470598 +0000 UTC m=+691.364091003" watchObservedRunningTime="2026-09-10 21:09:23.445500306 +0000 UTC m=+691.366120697"
Sep 10 21:09:23.458686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.458622    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" podStartSLOduration=1.4586068939999999 podStartE2EDuration="1.458606894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:23.458528815 +0000 UTC m=+691.379149207" watchObservedRunningTime="2026-09-10 21:09:23.458606894 +0000 UTC m=+691.379227280"
Sep 10 21:09:23.702111 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:23.701999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7746-4f0df7"]
Sep 10 21:09:23.704386 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:23.704359    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2a58b9f_914b_4807_9f2e_1ca514787528.slice/crio-0768571f69419a9d19cfbc384b6df034d53d4d063dea27e16fad891821c83505 WatchSource:0}: Error finding container 0768571f69419a9d19cfbc384b6df034d53d4d063dea27e16fad891821c83505: Status 404 returned error can't find the container with id 0768571f69419a9d19cfbc384b6df034d53d4d063dea27e16fad891821c83505
Sep 10 21:09:24.436057 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:24.436021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-4f0df7" event={"ID":"b2a58b9f-914b-4807-9f2e-1ca514787528","Type":"ContainerStarted","Data":"a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c"}
Sep 10 21:09:24.436531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:24.436073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7746-4f0df7" event={"ID":"b2a58b9f-914b-4807-9f2e-1ca514787528","Type":"ContainerStarted","Data":"0768571f69419a9d19cfbc384b6df034d53d4d063dea27e16fad891821c83505"}
Sep 10 21:09:24.436531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:24.436215    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7746-30d6f4" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" containerName="test-curl-7746-30d6f4" containerID="cri-o://96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05" gracePeriod=2
Sep 10 21:09:24.436696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:24.436595    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:24.450703 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:24.450642    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7746-4f0df7" podStartSLOduration=1.450622426 podStartE2EDuration="1.450622426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:24.450303907 +0000 UTC m=+692.370924296" watchObservedRunningTime="2026-09-10 21:09:24.450622426 +0000 UTC m=+692.371242819"
Sep 10 21:09:24.452317 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:24.452294    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-30d6f4\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:25.005833 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:25.005787    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7746-4f0df7"]
Sep 10 21:09:25.026174 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:25.026140    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7746-4f0df7"]
Sep 10 21:09:25.440623 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:25.440598    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7746-4f0df7" container="test-curl-7746-4f0df7"
Sep 10 21:09:26.444800 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:26.444765    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7746-4f0df7" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" containerName="test-curl-7746-4f0df7" containerID="cri-o://a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c" gracePeriod=2
Sep 10 21:09:26.447264 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:26.447238    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-4f0df7\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:26.607850 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:26.607827    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:26.610027 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:26.610002    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-4f0df7\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:26.637769 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:26.637745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsw6j\" (UniqueName: \"kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j\") pod \"56a8b17c-a060-49b5-9efc-bc43c14a515a\" (UID: \"56a8b17c-a060-49b5-9efc-bc43c14a515a\") "
Sep 10 21:09:26.639780 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:26.639759    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j" (OuterVolumeSpecName: "kube-api-access-vsw6j") pod "56a8b17c-a060-49b5-9efc-bc43c14a515a" (UID: "56a8b17c-a060-49b5-9efc-bc43c14a515a"). InnerVolumeSpecName "kube-api-access-vsw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:26.738489 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:26.738413    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsw6j\" (UniqueName: \"kubernetes.io/projected/56a8b17c-a060-49b5-9efc-bc43c14a515a-kube-api-access-vsw6j\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:09:27.450198 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:27.450165    2562 generic.go:356] "Generic (PLEG): container finished" podID="56a8b17c-a060-49b5-9efc-bc43c14a515a" containerID="96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05" exitCode=137
Sep 10 21:09:27.450618 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:27.450213    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-30d6f4"
Sep 10 21:09:27.450618 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:27.450244    2562 scope.go:117] "RemoveContainer" containerID="96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05"
Sep 10 21:09:27.458436 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:27.458389    2562 scope.go:117] "RemoveContainer" containerID="96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05"
Sep 10 21:09:27.458655 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:27.458633    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05\": container with ID starting with 96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05 not found: ID does not exist" containerID="96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05"
Sep 10 21:09:27.458717 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:27.458664    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05"} err="failed to get container status \"96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05\": rpc error: code = NotFound desc = could not find container \"96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05\": container with ID starting with 96329ea3403b9515696214fbd459a8c327feb87d0bf1fb54806b5172d49e7a05 not found: ID does not exist"
Sep 10 21:09:28.618928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:28.618895    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56a8b17c-a060-49b5-9efc-bc43c14a515a" path="/var/lib/kubelet/pods/56a8b17c-a060-49b5-9efc-bc43c14a515a/volumes"
Sep 10 21:09:28.638173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:28.638151    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:28.654210 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:28.654188    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrkf8\" (UniqueName: \"kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8\") pod \"b2a58b9f-914b-4807-9f2e-1ca514787528\" (UID: \"b2a58b9f-914b-4807-9f2e-1ca514787528\") "
Sep 10 21:09:28.656537 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:28.656515    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8" (OuterVolumeSpecName: "kube-api-access-wrkf8") pod "b2a58b9f-914b-4807-9f2e-1ca514787528" (UID: "b2a58b9f-914b-4807-9f2e-1ca514787528"). InnerVolumeSpecName "kube-api-access-wrkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:09:28.755505 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:28.755443    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkf8\" (UniqueName: \"kubernetes.io/projected/b2a58b9f-914b-4807-9f2e-1ca514787528-kube-api-access-wrkf8\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:09:29.461928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:29.461888    2562 generic.go:356] "Generic (PLEG): container finished" podID="b2a58b9f-914b-4807-9f2e-1ca514787528" containerID="a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c" exitCode=137
Sep 10 21:09:29.461928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:29.461927    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:29.462123 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:29.461992    2562 scope.go:117] "RemoveContainer" containerID="a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c"
Sep 10 21:09:29.470670 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:29.470652    2562 scope.go:117] "RemoveContainer" containerID="a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c"
Sep 10 21:09:29.470956 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:29.470936    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c\": container with ID starting with a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c not found: ID does not exist" containerID="a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c"
Sep 10 21:09:29.471024 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:29.470963    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c"} err="failed to get container status \"a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c\": rpc error: code = NotFound desc = could not find container \"a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c\": container with ID starting with a7098ed22185976d9930209847efa73938a3e5aed1d58d1677dfcac30e3f487c not found: ID does not exist"
Sep 10 21:09:29.472788 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:29.472760    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-4f0df7\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:30.036451 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.036415    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:09:30.036855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.036732    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" containerName="test-curl-7746-4f0df7"
Sep 10 21:09:30.036855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.036742    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" containerName="test-curl-7746-4f0df7"
Sep 10 21:09:30.036855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.036830    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" containerName="test-curl-7746-4f0df7"
Sep 10 21:09:30.041075 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.041058    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.043428 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:30.043402    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-4f0df7\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:30.043764 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.043749    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0827ee43\""
Sep 10 21:09:30.044669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.044654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0827ee43-dockercfg-d8jc2\""
Sep 10 21:09:30.053709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.053683    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:09:30.065759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.065735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.065877 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.065780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzh4\" (UniqueName: \"kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.121257 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.121229    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:09:30.124393 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.124378    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.132633 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.132603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:09:30.145397 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:09:30.145358    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7746-4f0df7\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" pod="opendatahub/test-curl-7746-4f0df7"
Sep 10 21:09:30.166802 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.166776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzh4\" (UniqueName: \"kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.166802 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.166826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjfd\" (UniqueName: \"kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.167014 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.166926    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.167063 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.167023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.167632 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.167610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.174575 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.174554    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzh4\" (UniqueName: \"kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4\") pod \"payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.267927 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.267893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.268088 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.267987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjfd\" (UniqueName: \"kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.268645 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.268626    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.276410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.276385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjfd\" (UniqueName: \"kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd\") pod \"payload-processing-e2e-ait-0827ee43-78c485b659-chws5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") " pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.351717 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.351635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:30.435290 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.435253    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:30.618182 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.618084    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a58b9f-914b-4807-9f2e-1ca514787528" path="/var/lib/kubelet/pods/b2a58b9f-914b-4807-9f2e-1ca514787528/volumes"
Sep 10 21:09:30.629845 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.628692    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:09:30.632455 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:30.632422    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bb2f69_a127_4acc_b22c_c47ab4844949.slice/crio-b7624f60125163203b2e3fb1dddc3839b95da3991abc9fc97fc30453231c6b70 WatchSource:0}: Error finding container b7624f60125163203b2e3fb1dddc3839b95da3991abc9fc97fc30453231c6b70: Status 404 returned error can't find the container with id b7624f60125163203b2e3fb1dddc3839b95da3991abc9fc97fc30453231c6b70
Sep 10 21:09:30.712367 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:30.712334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:09:30.714547 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:30.714506    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7f0824_c293_441f_bad0_2c170e78eec5.slice/crio-5a946c90c870bd0dfeaabf52d34b5a6465dbc060a354f7f45358cadf3cc5e95e WatchSource:0}: Error finding container 5a946c90c870bd0dfeaabf52d34b5a6465dbc060a354f7f45358cadf3cc5e95e: Status 404 returned error can't find the container with id 5a946c90c870bd0dfeaabf52d34b5a6465dbc060a354f7f45358cadf3cc5e95e
Sep 10 21:09:31.157495 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.157459    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:09:31.161168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.161152    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.163936 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.163909    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-94604f-dockercfg-lq6r8\""
Sep 10 21:09:31.164064 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.164046    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-94604f\""
Sep 10 21:09:31.170640 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.170617    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:09:31.176275 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.176249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmgd\" (UniqueName: \"kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.176375 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.176363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.223780 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.223749    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:09:31.227333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.227313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.236193 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.236169    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:09:31.277549 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.277520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s98\" (UniqueName: \"kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.277684 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.277570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.277684 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.277611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.277757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.277739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmgd\" (UniqueName: \"kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.278368 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.278350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.285763 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.285743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmgd\" (UniqueName: \"kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd\") pod \"payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.379153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.379116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s98\" (UniqueName: \"kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.379307 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.379198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.379925 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.379903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.391697 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.391675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s98\" (UniqueName: \"kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98\") pod \"payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") " pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.472403 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.472361    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:31.474131 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.474106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" event={"ID":"5f7f0824-c293-441f-bad0-2c170e78eec5","Type":"ContainerStarted","Data":"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"}
Sep 10 21:09:31.474270 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.474153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" event={"ID":"5f7f0824-c293-441f-bad0-2c170e78eec5","Type":"ContainerStarted","Data":"5a946c90c870bd0dfeaabf52d34b5a6465dbc060a354f7f45358cadf3cc5e95e"}
Sep 10 21:09:31.474270 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.474227    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:31.475908 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.475884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" event={"ID":"b2bb2f69-a127-4acc-b22c-c47ab4844949","Type":"ContainerStarted","Data":"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"}
Sep 10 21:09:31.475991 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.475927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" event={"ID":"b2bb2f69-a127-4acc-b22c-c47ab4844949","Type":"ContainerStarted","Data":"b7624f60125163203b2e3fb1dddc3839b95da3991abc9fc97fc30453231c6b70"}
Sep 10 21:09:31.475991 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.475956    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:31.498732 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.498685    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" podStartSLOduration=1.498669128 podStartE2EDuration="1.498669128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:31.497533856 +0000 UTC m=+699.418154261" watchObservedRunningTime="2026-09-10 21:09:31.498669128 +0000 UTC m=+699.419289514"
Sep 10 21:09:31.517169 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.517118    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" podStartSLOduration=1.5171035210000001 podStartE2EDuration="1.517103521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:31.515082586 +0000 UTC m=+699.435702973" watchObservedRunningTime="2026-09-10 21:09:31.517103521 +0000 UTC m=+699.437723906"
Sep 10 21:09:31.538355 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.538332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:31.744559 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.744529    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:09:31.746379 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:31.746353    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fd7131_f4ea_4b78_952b_1039869776a2.slice/crio-e692c52b3f27f103eb3afab164dd018b270aeaa1956dea3cdb573bd10512547c WatchSource:0}: Error finding container e692c52b3f27f103eb3afab164dd018b270aeaa1956dea3cdb573bd10512547c: Status 404 returned error can't find the container with id e692c52b3f27f103eb3afab164dd018b270aeaa1956dea3cdb573bd10512547c
Sep 10 21:09:31.822992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:31.822971    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:09:31.825051 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:31.825023    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6349fe1f_6ec6_4fe2_bd09_0e5ae84a32d8.slice/crio-8e1c7723b2065d3f1d15f3a197c9ba75792f1752cb2a5ea623d9e05c43ab1115 WatchSource:0}: Error finding container 8e1c7723b2065d3f1d15f3a197c9ba75792f1752cb2a5ea623d9e05c43ab1115: Status 404 returned error can't find the container with id 8e1c7723b2065d3f1d15f3a197c9ba75792f1752cb2a5ea623d9e05c43ab1115
Sep 10 21:09:32.484194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.484146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" event={"ID":"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8","Type":"ContainerStarted","Data":"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"}
Sep 10 21:09:32.484777 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.484218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" event={"ID":"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8","Type":"ContainerStarted","Data":"8e1c7723b2065d3f1d15f3a197c9ba75792f1752cb2a5ea623d9e05c43ab1115"}
Sep 10 21:09:32.484777 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.484252    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:32.486562 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.486538    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" event={"ID":"d0fd7131-f4ea-4b78-952b-1039869776a2","Type":"ContainerStarted","Data":"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"}
Sep 10 21:09:32.486752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.486726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" event={"ID":"d0fd7131-f4ea-4b78-952b-1039869776a2","Type":"ContainerStarted","Data":"e692c52b3f27f103eb3afab164dd018b270aeaa1956dea3cdb573bd10512547c"}
Sep 10 21:09:32.503027 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.502975    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" podStartSLOduration=1.502957257 podStartE2EDuration="1.502957257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:32.500785673 +0000 UTC m=+700.421406075" watchObservedRunningTime="2026-09-10 21:09:32.502957257 +0000 UTC m=+700.423577647"
Sep 10 21:09:32.516641 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:32.516599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" podStartSLOduration=1.516586183 podStartE2EDuration="1.516586183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:32.51558543 +0000 UTC m=+700.436205827" watchObservedRunningTime="2026-09-10 21:09:32.516586183 +0000 UTC m=+700.437206597"
Sep 10 21:09:33.492285 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:33.492256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:34.438246 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:34.438218    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:09:35.442422 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:35.442395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:09:42.488535 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:42.488491    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:09:42.488979 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:42.488577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:09:43.494217 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:43.494183    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:09:44.499751 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:44.499719    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:09:50.622939 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.622866    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:09:50.626100 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.626079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.628707 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.628687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-69dcc8\""
Sep 10 21:09:50.628834 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.628776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-69dcc8-dockercfg-c8vmw\""
Sep 10 21:09:50.640195 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.640168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:09:50.741145 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.741115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpg4\" (UniqueName: \"kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.741283 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.741154    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.841986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.841952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpg4\" (UniqueName: \"kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.842133 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.841992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.842710 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.842686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.849493 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.849465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpg4\" (UniqueName: \"kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4\") pod \"payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:50.936551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:50.936475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:51.204651 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:51.204626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:09:51.206747 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:09:51.206720    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd993c51_32c7_44fc_8954_18a4017cd94c.slice/crio-e892e8c77b885b875e4c619ce495c725568522ebddc383a77473762e32cbe93b WatchSource:0}: Error finding container e892e8c77b885b875e4c619ce495c725568522ebddc383a77473762e32cbe93b: Status 404 returned error can't find the container with id e892e8c77b885b875e4c619ce495c725568522ebddc383a77473762e32cbe93b
Sep 10 21:09:51.580086 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:51.579995    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" event={"ID":"fd993c51-32c7-44fc-8954-18a4017cd94c","Type":"ContainerStarted","Data":"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"}
Sep 10 21:09:51.580086 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:51.580055    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" event={"ID":"fd993c51-32c7-44fc-8954-18a4017cd94c","Type":"ContainerStarted","Data":"e892e8c77b885b875e4c619ce495c725568522ebddc383a77473762e32cbe93b"}
Sep 10 21:09:51.580300 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:51.580093    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:09:51.597033 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:09:51.596988    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" podStartSLOduration=1.596973213 podStartE2EDuration="1.596973213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:51.59553997 +0000 UTC m=+719.516160359" watchObservedRunningTime="2026-09-10 21:09:51.596973213 +0000 UTC m=+719.517593599"
Sep 10 21:10:02.586466 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:02.586435    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:10:09.954468 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:09.954436    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:10:09.954969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:09.954670    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" podUID="5f7f0824-c293-441f-bad0-2c170e78eec5" containerName="payload-processing" containerID="cri-o://bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9" gracePeriod=30
Sep 10 21:10:09.983153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:09.983124    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:10:09.983389 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:09.983362    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" podUID="b2bb2f69-a127-4acc-b22c-c47ab4844949" containerName="payload-pre-processing" containerID="cri-o://dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a" gracePeriod=30
Sep 10 21:10:10.240745 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.240722    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:10:10.243215 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.243180    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:10:10.409961 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.409932    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dzh4\" (UniqueName: \"kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4\") pod \"b2bb2f69-a127-4acc-b22c-c47ab4844949\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") "
Sep 10 21:10:10.410132 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.409980    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume\") pod \"5f7f0824-c293-441f-bad0-2c170e78eec5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") "
Sep 10 21:10:10.410132 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.410030    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume\") pod \"b2bb2f69-a127-4acc-b22c-c47ab4844949\" (UID: \"b2bb2f69-a127-4acc-b22c-c47ab4844949\") "
Sep 10 21:10:10.410132 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.410083    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdjfd\" (UniqueName: \"kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd\") pod \"5f7f0824-c293-441f-bad0-2c170e78eec5\" (UID: \"5f7f0824-c293-441f-bad0-2c170e78eec5\") "
Sep 10 21:10:10.410455 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.410425    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f7f0824-c293-441f-bad0-2c170e78eec5" (UID: "5f7f0824-c293-441f-bad0-2c170e78eec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:10.410455 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.410436    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2bb2f69-a127-4acc-b22c-c47ab4844949" (UID: "b2bb2f69-a127-4acc-b22c-c47ab4844949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:10.412265 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.412238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4" (OuterVolumeSpecName: "kube-api-access-9dzh4") pod "b2bb2f69-a127-4acc-b22c-c47ab4844949" (UID: "b2bb2f69-a127-4acc-b22c-c47ab4844949"). InnerVolumeSpecName "kube-api-access-9dzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:10.414099 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.414076    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd" (OuterVolumeSpecName: "kube-api-access-pdjfd") pod "5f7f0824-c293-441f-bad0-2c170e78eec5" (UID: "5f7f0824-c293-441f-bad0-2c170e78eec5"). InnerVolumeSpecName "kube-api-access-pdjfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:10.510750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.510691    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2bb2f69-a127-4acc-b22c-c47ab4844949-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:10.510750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.510713    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdjfd\" (UniqueName: \"kubernetes.io/projected/5f7f0824-c293-441f-bad0-2c170e78eec5-kube-api-access-pdjfd\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:10.510750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.510722    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzh4\" (UniqueName: \"kubernetes.io/projected/b2bb2f69-a127-4acc-b22c-c47ab4844949-kube-api-access-9dzh4\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:10.510750 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.510732    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f7f0824-c293-441f-bad0-2c170e78eec5-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:10.663222 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.663192    2562 generic.go:356] "Generic (PLEG): container finished" podID="5f7f0824-c293-441f-bad0-2c170e78eec5" containerID="bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9" exitCode=0
Sep 10 21:10:10.663332 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.663238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"
Sep 10 21:10:10.663332 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.663279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" event={"ID":"5f7f0824-c293-441f-bad0-2c170e78eec5","Type":"ContainerDied","Data":"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"}
Sep 10 21:10:10.663421 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.663335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5" event={"ID":"5f7f0824-c293-441f-bad0-2c170e78eec5","Type":"ContainerDied","Data":"5a946c90c870bd0dfeaabf52d34b5a6465dbc060a354f7f45358cadf3cc5e95e"}
Sep 10 21:10:10.663421 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.663366    2562 scope.go:117] "RemoveContainer" containerID="bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"
Sep 10 21:10:10.664558 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.664538    2562 generic.go:356] "Generic (PLEG): container finished" podID="b2bb2f69-a127-4acc-b22c-c47ab4844949" containerID="dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a" exitCode=0
Sep 10 21:10:10.664624 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.664574    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"
Sep 10 21:10:10.664624 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.664595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" event={"ID":"b2bb2f69-a127-4acc-b22c-c47ab4844949","Type":"ContainerDied","Data":"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"}
Sep 10 21:10:10.664701 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.664631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf" event={"ID":"b2bb2f69-a127-4acc-b22c-c47ab4844949","Type":"ContainerDied","Data":"b7624f60125163203b2e3fb1dddc3839b95da3991abc9fc97fc30453231c6b70"}
Sep 10 21:10:10.672072 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.672052    2562 scope.go:117] "RemoveContainer" containerID="bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"
Sep 10 21:10:10.672345 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:10:10.672326    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9\": container with ID starting with bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9 not found: ID does not exist" containerID="bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"
Sep 10 21:10:10.672417 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.672357    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9"} err="failed to get container status \"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9\": rpc error: code = NotFound desc = could not find container \"bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9\": container with ID starting with bf7451e075d333d9499dd07169d77eb2d53fc767a283492815a5bc33611f22e9 not found: ID does not exist"
Sep 10 21:10:10.672417 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.672380    2562 scope.go:117] "RemoveContainer" containerID="dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"
Sep 10 21:10:10.679606 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.679592    2562 scope.go:117] "RemoveContainer" containerID="dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"
Sep 10 21:10:10.679927 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:10:10.679899    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a\": container with ID starting with dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a not found: ID does not exist" containerID="dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"
Sep 10 21:10:10.679986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.679938    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a"} err="failed to get container status \"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a\": rpc error: code = NotFound desc = could not find container \"dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a\": container with ID starting with dcad8a82501aa0754749b848d4316710336ddc72c519801f6d09ff59b51d132a not found: ID does not exist"
Sep 10 21:10:10.680679 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.680661    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:10:10.685387 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.685366    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0827ee43-78c485b659-chws5"]
Sep 10 21:10:10.697846 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.697827    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:10:10.703767 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:10.703746    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0827ee43-78b67bfd6b-tlmkf"]
Sep 10 21:10:12.618297 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:12.618266    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f7f0824-c293-441f-bad0-2c170e78eec5" path="/var/lib/kubelet/pods/5f7f0824-c293-441f-bad0-2c170e78eec5/volumes"
Sep 10 21:10:12.618725 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:12.618697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2bb2f69-a127-4acc-b22c-c47ab4844949" path="/var/lib/kubelet/pods/b2bb2f69-a127-4acc-b22c-c47ab4844949/volumes"
Sep 10 21:10:15.298360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298328    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:15.298709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298669    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f7f0824-c293-441f-bad0-2c170e78eec5" containerName="payload-processing"
Sep 10 21:10:15.298709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298684    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f7f0824-c293-441f-bad0-2c170e78eec5" containerName="payload-processing"
Sep 10 21:10:15.298709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298699    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2bb2f69-a127-4acc-b22c-c47ab4844949" containerName="payload-pre-processing"
Sep 10 21:10:15.298709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298704    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2bb2f69-a127-4acc-b22c-c47ab4844949" containerName="payload-pre-processing"
Sep 10 21:10:15.298869 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298774    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2bb2f69-a127-4acc-b22c-c47ab4844949" containerName="payload-pre-processing"
Sep 10 21:10:15.298869 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.298782    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f7f0824-c293-441f-bad0-2c170e78eec5" containerName="payload-processing"
Sep 10 21:10:15.303110 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.303088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.305857 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.305836    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c2ec79e5-dockercfg-gsmgw\""
Sep 10 21:10:15.305983 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.305840    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c2ec79e5\""
Sep 10 21:10:15.313631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.313610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:15.449255 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.449218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.449407 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.449368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slf9x\" (UniqueName: \"kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.550372 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.550272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slf9x\" (UniqueName: \"kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.550372 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.550333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.551057 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.551035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.558087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.558070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slf9x\" (UniqueName: \"kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x\") pod \"payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") " pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.613862 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.613830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:15.886052 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:15.886026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:15.888337 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:10:15.888310    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85464b0_2912_4c0d_9106_f03dadcbe369.slice/crio-a3c02d06ef9de23968e0df3d9a2141bc56a7330fa33a0a70ea9640af1cd27315 WatchSource:0}: Error finding container a3c02d06ef9de23968e0df3d9a2141bc56a7330fa33a0a70ea9640af1cd27315: Status 404 returned error can't find the container with id a3c02d06ef9de23968e0df3d9a2141bc56a7330fa33a0a70ea9640af1cd27315
Sep 10 21:10:16.696446 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:16.696409    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" event={"ID":"a85464b0-2912-4c0d-9106-f03dadcbe369","Type":"ContainerStarted","Data":"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"}
Sep 10 21:10:16.696856 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:16.696468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" event={"ID":"a85464b0-2912-4c0d-9106-f03dadcbe369","Type":"ContainerStarted","Data":"a3c02d06ef9de23968e0df3d9a2141bc56a7330fa33a0a70ea9640af1cd27315"}
Sep 10 21:10:16.696856 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:16.696495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:16.712671 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:16.712623    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" podStartSLOduration=1.712609235 podStartE2EDuration="1.712609235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:16.711470581 +0000 UTC m=+744.632090993" watchObservedRunningTime="2026-09-10 21:10:16.712609235 +0000 UTC m=+744.633229621"
Sep 10 21:10:19.644630 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.644595    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:10:19.647975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.647959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.650368 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.650351    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-267aa8ed-kserve-self-signed-certs\""
Sep 10 21:10:19.655699 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.655679    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:10:19.785021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.784994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.785161 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.785035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.785161 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.785132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.785161 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.785159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.785292 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.785277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.785341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.785302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfx9\" (UniqueName: \"kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886389 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfx9\" (UniqueName: \"kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886718 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886860 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.886935 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.886837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.887030 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.887014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.888786 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.888766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.889067 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.889047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.894756 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.894709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfx9\" (UniqueName: \"kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9\") pod \"e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") " pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:19.957095 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:19.957072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:20.223844 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:20.223820    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:10:20.226148 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:10:20.226121    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f953e3_73f5_4824_aa95_09b33c24efad.slice/crio-552d32764304888db1a1dba13aa8671bba0f9060d6ac8a6d2de6682cdd3442f9 WatchSource:0}: Error finding container 552d32764304888db1a1dba13aa8671bba0f9060d6ac8a6d2de6682cdd3442f9: Status 404 returned error can't find the container with id 552d32764304888db1a1dba13aa8671bba0f9060d6ac8a6d2de6682cdd3442f9
Sep 10 21:10:20.720018 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:20.719983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" event={"ID":"51f953e3-73f5-4824-aa95-09b33c24efad","Type":"ContainerStarted","Data":"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"}
Sep 10 21:10:20.720018 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:20.720022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" event={"ID":"51f953e3-73f5-4824-aa95-09b33c24efad","Type":"ContainerStarted","Data":"552d32764304888db1a1dba13aa8671bba0f9060d6ac8a6d2de6682cdd3442f9"}
Sep 10 21:10:20.736307 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:20.736264    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" podStartSLOduration=1.7362508920000002 podStartE2EDuration="1.736250892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:20.735327907 +0000 UTC m=+748.655948311" watchObservedRunningTime="2026-09-10 21:10:20.736250892 +0000 UTC m=+748.656871305"
Sep 10 21:10:27.708211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:27.708178    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:29.957638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:29.957601    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:29.958147 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:29.957678    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:29.970216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:29.970194    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:30.778056 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:30.778025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:10:45.260858 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.260783    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:10:45.265226 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.265210    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.269192 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.269164    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a41a24\"/\"kube-root-ca.crt\""
Sep 10 21:10:45.269192 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.269167    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a41a24\"/\"test-model-a41a24-kserve-self-signed-certs\""
Sep 10 21:10:45.269192 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.269206    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a41a24\"/\"openshift-service-ca.crt\""
Sep 10 21:10:45.269488 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.269167    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a41a24\"/\"default-dockercfg-ngwft\""
Sep 10 21:10:45.274870 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.274850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:10:45.298817 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.298786    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f54h\" (UniqueName: \"kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.298923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.298851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.298970 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.298932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.299005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.298970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.299005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.298985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.299078 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.299007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400272 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f54h\" (UniqueName: \"kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400410 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400517 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400562 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400765 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400744    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400850 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.400850 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.400844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.402619 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.402595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.402799 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.402785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.408635 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.408613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f54h\" (UniqueName: \"kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h\") pod \"test-model-a41a24-kserve-745dfb5867-9jnlq\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") " pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.575856 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.575760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:45.843597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:45.843578    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:10:45.845783 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:10:45.845754    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ef4b3e0_05a3_4c62_a66c_eb938eed7ef0.slice/crio-1ec10df4416fccd4127c7a445f71422f9658b2388192046a8691342eeb6903a2 WatchSource:0}: Error finding container 1ec10df4416fccd4127c7a445f71422f9658b2388192046a8691342eeb6903a2: Status 404 returned error can't find the container with id 1ec10df4416fccd4127c7a445f71422f9658b2388192046a8691342eeb6903a2
Sep 10 21:10:46.149229 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.149155    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:46.149375 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.149346    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" podUID="a85464b0-2912-4c0d-9106-f03dadcbe369" containerName="payload-processing" containerID="cri-o://fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2" gracePeriod=30
Sep 10 21:10:46.386933 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.386909    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:46.509656 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.509623    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume\") pod \"a85464b0-2912-4c0d-9106-f03dadcbe369\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") "
Sep 10 21:10:46.509821 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.509664    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slf9x\" (UniqueName: \"kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x\") pod \"a85464b0-2912-4c0d-9106-f03dadcbe369\" (UID: \"a85464b0-2912-4c0d-9106-f03dadcbe369\") "
Sep 10 21:10:46.509967 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.509947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a85464b0-2912-4c0d-9106-f03dadcbe369" (UID: "a85464b0-2912-4c0d-9106-f03dadcbe369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:46.511832 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.511776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x" (OuterVolumeSpecName: "kube-api-access-slf9x") pod "a85464b0-2912-4c0d-9106-f03dadcbe369" (UID: "a85464b0-2912-4c0d-9106-f03dadcbe369"). InnerVolumeSpecName "kube-api-access-slf9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:46.610545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.610513    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a85464b0-2912-4c0d-9106-f03dadcbe369-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:46.610545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.610539    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slf9x\" (UniqueName: \"kubernetes.io/projected/a85464b0-2912-4c0d-9106-f03dadcbe369-kube-api-access-slf9x\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:10:46.838169 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.838134    2562 generic.go:356] "Generic (PLEG): container finished" podID="a85464b0-2912-4c0d-9106-f03dadcbe369" containerID="fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2" exitCode=0
Sep 10 21:10:46.838381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.838195    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"
Sep 10 21:10:46.838381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.838221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" event={"ID":"a85464b0-2912-4c0d-9106-f03dadcbe369","Type":"ContainerDied","Data":"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"}
Sep 10 21:10:46.838381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.838294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v" event={"ID":"a85464b0-2912-4c0d-9106-f03dadcbe369","Type":"ContainerDied","Data":"a3c02d06ef9de23968e0df3d9a2141bc56a7330fa33a0a70ea9640af1cd27315"}
Sep 10 21:10:46.838381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.838322    2562 scope.go:117] "RemoveContainer" containerID="fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"
Sep 10 21:10:46.839955 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.839927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" event={"ID":"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0","Type":"ContainerStarted","Data":"d2c59a338270ea72180ebd4b1bd4adb7bffa72ec4703be5caf4ba92f10ddb04f"}
Sep 10 21:10:46.840078 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.839961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" event={"ID":"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0","Type":"ContainerStarted","Data":"1ec10df4416fccd4127c7a445f71422f9658b2388192046a8691342eeb6903a2"}
Sep 10 21:10:46.846675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.846655    2562 scope.go:117] "RemoveContainer" containerID="fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"
Sep 10 21:10:46.846963 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:10:46.846944    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2\": container with ID starting with fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2 not found: ID does not exist" containerID="fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"
Sep 10 21:10:46.847053 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.846969    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2"} err="failed to get container status \"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2\": rpc error: code = NotFound desc = could not find container \"fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2\": container with ID starting with fce808472488a89b23e241a8ecba15e0607b3184f8039339ccf0598c614ddef2 not found: ID does not exist"
Sep 10 21:10:46.860868 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.860828    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" podStartSLOduration=1.860800227 podStartE2EDuration="1.860800227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:46.858861351 +0000 UTC m=+774.779481757" watchObservedRunningTime="2026-09-10 21:10:46.860800227 +0000 UTC m=+774.781420613"
Sep 10 21:10:46.871915 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.871894    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:46.877191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:46.877173    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c2ec79e5-79db788686-t7j2v"]
Sep 10 21:10:48.619737 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:48.619707    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85464b0-2912-4c0d-9106-f03dadcbe369" path="/var/lib/kubelet/pods/a85464b0-2912-4c0d-9106-f03dadcbe369/volumes"
Sep 10 21:10:49.243216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:49.243186    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:10:55.576843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:55.576791    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:55.576843 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:55.576850    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:55.589463 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:55.589438    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:10:55.893226 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:10:55.893145    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:11:02.541975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.541937    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:11:02.542438 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.542192    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" podUID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" containerName="payload-processing" containerID="cri-o://135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc" gracePeriod=30
Sep 10 21:11:02.549555 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.549530    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:11:02.549795 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.549738    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" podUID="d0fd7131-f4ea-4b78-952b-1039869776a2" containerName="payload-pre-processing" containerID="cri-o://debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2" gracePeriod=30
Sep 10 21:11:02.812117 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.812097    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:11:02.814607 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.814591    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:11:02.915794 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.915759    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0fd7131-f4ea-4b78-952b-1039869776a2" containerID="debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2" exitCode=0
Sep 10 21:11:02.915973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.915825    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"
Sep 10 21:11:02.915973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.915857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" event={"ID":"d0fd7131-f4ea-4b78-952b-1039869776a2","Type":"ContainerDied","Data":"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"}
Sep 10 21:11:02.915973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.915905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp" event={"ID":"d0fd7131-f4ea-4b78-952b-1039869776a2","Type":"ContainerDied","Data":"e692c52b3f27f103eb3afab164dd018b270aeaa1956dea3cdb573bd10512547c"}
Sep 10 21:11:02.915973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.915931    2562 scope.go:117] "RemoveContainer" containerID="debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"
Sep 10 21:11:02.917686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.917655    2562 generic.go:356] "Generic (PLEG): container finished" podID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" containerID="135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc" exitCode=0
Sep 10 21:11:02.917776 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.917729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" event={"ID":"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8","Type":"ContainerDied","Data":"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"}
Sep 10 21:11:02.917776 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.917766    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp" event={"ID":"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8","Type":"ContainerDied","Data":"8e1c7723b2065d3f1d15f3a197c9ba75792f1752cb2a5ea623d9e05c43ab1115"}
Sep 10 21:11:02.917988 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.917840    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"
Sep 10 21:11:02.925496 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.925480    2562 scope.go:117] "RemoveContainer" containerID="debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"
Sep 10 21:11:02.925748 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:11:02.925730    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2\": container with ID starting with debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2 not found: ID does not exist" containerID="debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"
Sep 10 21:11:02.925791 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.925757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2"} err="failed to get container status \"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2\": rpc error: code = NotFound desc = could not find container \"debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2\": container with ID starting with debed69f36060aeb99f0cfcf488b2444d6abce49a783f6efb787aa063f0901d2 not found: ID does not exist"
Sep 10 21:11:02.925791 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.925772    2562 scope.go:117] "RemoveContainer" containerID="135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"
Sep 10 21:11:02.933076 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.933060    2562 scope.go:117] "RemoveContainer" containerID="135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"
Sep 10 21:11:02.933335 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:11:02.933319    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc\": container with ID starting with 135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc not found: ID does not exist" containerID="135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"
Sep 10 21:11:02.933376 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.933342    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc"} err="failed to get container status \"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc\": rpc error: code = NotFound desc = could not find container \"135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc\": container with ID starting with 135f370d6c82df3ed05a1b01a7f6e1920b8806b3ff0b99b05286ae0a53df6fdc not found: ID does not exist"
Sep 10 21:11:02.950676 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.950654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtmgd\" (UniqueName: \"kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd\") pod \"d0fd7131-f4ea-4b78-952b-1039869776a2\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") "
Sep 10 21:11:02.950740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.950691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4s98\" (UniqueName: \"kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98\") pod \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") "
Sep 10 21:11:02.950740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.950715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume\") pod \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\" (UID: \"6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8\") "
Sep 10 21:11:02.950740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.950738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume\") pod \"d0fd7131-f4ea-4b78-952b-1039869776a2\" (UID: \"d0fd7131-f4ea-4b78-952b-1039869776a2\") "
Sep 10 21:11:02.951068 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.951045    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" (UID: "6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:02.951147 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.951102    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0fd7131-f4ea-4b78-952b-1039869776a2" (UID: "d0fd7131-f4ea-4b78-952b-1039869776a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:02.952669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.952648    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd" (OuterVolumeSpecName: "kube-api-access-wtmgd") pod "d0fd7131-f4ea-4b78-952b-1039869776a2" (UID: "d0fd7131-f4ea-4b78-952b-1039869776a2"). InnerVolumeSpecName "kube-api-access-wtmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:02.952733 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:02.952708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98" (OuterVolumeSpecName: "kube-api-access-n4s98") pod "6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" (UID: "6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8"). InnerVolumeSpecName "kube-api-access-n4s98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:03.051749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.051671    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmgd\" (UniqueName: \"kubernetes.io/projected/d0fd7131-f4ea-4b78-952b-1039869776a2-kube-api-access-wtmgd\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:03.051749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.051699    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4s98\" (UniqueName: \"kubernetes.io/projected/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-kube-api-access-n4s98\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:03.051749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.051709    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:03.051749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.051720    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0fd7131-f4ea-4b78-952b-1039869776a2-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:03.244083 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.244049    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:11:03.246085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.246062    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-94604f-59b4cbff67-x9wtp"]
Sep 10 21:11:03.256486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.256464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:11:03.262658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:03.262635    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-94604f-5848d7c4cd-knqfp"]
Sep 10 21:11:04.263085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.263044    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" podUID="51f953e3-73f5-4824-aa95-09b33c24efad" containerName="main" containerID="cri-o://ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c" gracePeriod=45
Sep 10 21:11:04.511744 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.511717    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:11:04.619538 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.619458    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" path="/var/lib/kubelet/pods/6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8/volumes"
Sep 10 21:11:04.619839 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.619825    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fd7131-f4ea-4b78-952b-1039869776a2" path="/var/lib/kubelet/pods/d0fd7131-f4ea-4b78-952b-1039869776a2/volumes"
Sep 10 21:11:04.664791 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664760    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.664969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.664969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664903    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.664969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664920    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpfx9\" (UniqueName: \"kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.664969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.664969 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.664969    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home\") pod \"51f953e3-73f5-4824-aa95-09b33c24efad\" (UID: \"51f953e3-73f5-4824-aa95-09b33c24efad\") "
Sep 10 21:11:04.665253 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.665213    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache" (OuterVolumeSpecName: "model-cache") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:11:04.665253 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.665225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:11:04.665253 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.665238    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home" (OuterVolumeSpecName: "home") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:11:04.667087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.667058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm" (OuterVolumeSpecName: "dshm") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:11:04.667087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.667053    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:11:04.667227 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.667155    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9" (OuterVolumeSpecName: "kube-api-access-bpfx9") pod "51f953e3-73f5-4824-aa95-09b33c24efad" (UID: "51f953e3-73f5-4824-aa95-09b33c24efad"). InnerVolumeSpecName "kube-api-access-bpfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:04.766442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766399    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.766442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766433    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51f953e3-73f5-4824-aa95-09b33c24efad-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.766442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766443    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.766442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766452    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfx9\" (UniqueName: \"kubernetes.io/projected/51f953e3-73f5-4824-aa95-09b33c24efad-kube-api-access-bpfx9\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.766696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766461    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.766696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.766470    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51f953e3-73f5-4824-aa95-09b33c24efad-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:11:04.930083 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.930002    2562 generic.go:356] "Generic (PLEG): container finished" podID="51f953e3-73f5-4824-aa95-09b33c24efad" containerID="ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c" exitCode=0
Sep 10 21:11:04.930083 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.930040    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"
Sep 10 21:11:04.930269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.930083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" event={"ID":"51f953e3-73f5-4824-aa95-09b33c24efad","Type":"ContainerDied","Data":"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"}
Sep 10 21:11:04.930269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.930133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42" event={"ID":"51f953e3-73f5-4824-aa95-09b33c24efad","Type":"ContainerDied","Data":"552d32764304888db1a1dba13aa8671bba0f9060d6ac8a6d2de6682cdd3442f9"}
Sep 10 21:11:04.930269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.930154    2562 scope.go:117] "RemoveContainer" containerID="ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"
Sep 10 21:11:04.939710 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.939693    2562 scope.go:117] "RemoveContainer" containerID="ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"
Sep 10 21:11:04.940024 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:11:04.939999    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c\": container with ID starting with ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c not found: ID does not exist" containerID="ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"
Sep 10 21:11:04.940103 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.940037    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c"} err="failed to get container status \"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c\": rpc error: code = NotFound desc = could not find container \"ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c\": container with ID starting with ca0d1c2dcef1e9751270aab148f96a0d08e0d9ce1cd062c28202c221cf44605c not found: ID does not exist"
Sep 10 21:11:04.953408 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.953381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:11:04.955145 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:04.955126    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-267aa8ed-kserve-5b95bfd5f-7gw42"]
Sep 10 21:11:06.618511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:06.618475    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f953e3-73f5-4824-aa95-09b33c24efad" path="/var/lib/kubelet/pods/51f953e3-73f5-4824-aa95-09b33c24efad/volumes"
Sep 10 21:11:15.140441 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140359    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:11:15.140894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140826    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fd7131-f4ea-4b78-952b-1039869776a2" containerName="payload-pre-processing"
Sep 10 21:11:15.140894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140845    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fd7131-f4ea-4b78-952b-1039869776a2" containerName="payload-pre-processing"
Sep 10 21:11:15.140894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f953e3-73f5-4824-aa95-09b33c24efad" containerName="main"
Sep 10 21:11:15.140894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140881    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f953e3-73f5-4824-aa95-09b33c24efad" containerName="main"
Sep 10 21:11:15.140894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140895    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85464b0-2912-4c0d-9106-f03dadcbe369" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140904    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85464b0-2912-4c0d-9106-f03dadcbe369" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140914    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.140922    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.141002    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6349fe1f-6ec6-4fe2-bd09-0e5ae84a32d8" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.141017    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fd7131-f4ea-4b78-952b-1039869776a2" containerName="payload-pre-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.141028    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85464b0-2912-4c0d-9106-f03dadcbe369" containerName="payload-processing"
Sep 10 21:11:15.141154 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.141037    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f953e3-73f5-4824-aa95-09b33c24efad" containerName="main"
Sep 10 21:11:15.145749 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.145730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.148535 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.148510    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-50ba47-dockercfg-fl6rx\""
Sep 10 21:11:15.148639 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.148516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-50ba47\""
Sep 10 21:11:15.156859 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.156837    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:11:15.256334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.256302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.256485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.256349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fbp\" (UniqueName: \"kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.357498 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.357465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.357681 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.357514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fbp\" (UniqueName: \"kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.358199 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.358181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.365905 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.365876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fbp\" (UniqueName: \"kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp\") pod \"payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") " pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.457030 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.456998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:15.724722 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.724691    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:11:15.726596 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:11:15.726568    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19581646_e9ee_48ee_9caf_16c4c4673480.slice/crio-5a5023a6628dffd8b1b5e85473409a8bcdeb6627b23fbf4242d987f12a98d5d8 WatchSource:0}: Error finding container 5a5023a6628dffd8b1b5e85473409a8bcdeb6627b23fbf4242d987f12a98d5d8: Status 404 returned error can't find the container with id 5a5023a6628dffd8b1b5e85473409a8bcdeb6627b23fbf4242d987f12a98d5d8
Sep 10 21:11:15.986949 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.986856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" event={"ID":"19581646-e9ee-48ee-9caf-16c4c4673480","Type":"ContainerStarted","Data":"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"}
Sep 10 21:11:15.986949 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.986917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" event={"ID":"19581646-e9ee-48ee-9caf-16c4c4673480","Type":"ContainerStarted","Data":"5a5023a6628dffd8b1b5e85473409a8bcdeb6627b23fbf4242d987f12a98d5d8"}
Sep 10 21:11:15.987165 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:15.986971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:11:16.005238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:16.005175    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" podStartSLOduration=1.005162778 podStartE2EDuration="1.005162778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:16.004634861 +0000 UTC m=+803.925255264" watchObservedRunningTime="2026-09-10 21:11:16.005162778 +0000 UTC m=+803.925783165"
Sep 10 21:11:26.994160 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:11:26.994132    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:13:01.500345 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.500307    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:13:01.502761 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.501039    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" podUID="19581646-e9ee-48ee-9caf-16c4c4673480" containerName="payload-processing" containerID="cri-o://f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2" gracePeriod=30
Sep 10 21:13:01.748081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.748051    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:13:01.849506 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.849413    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9fbp\" (UniqueName: \"kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp\") pod \"19581646-e9ee-48ee-9caf-16c4c4673480\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") "
Sep 10 21:13:01.849506 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.849501    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume\") pod \"19581646-e9ee-48ee-9caf-16c4c4673480\" (UID: \"19581646-e9ee-48ee-9caf-16c4c4673480\") "
Sep 10 21:13:01.849894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.849865    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19581646-e9ee-48ee-9caf-16c4c4673480" (UID: "19581646-e9ee-48ee-9caf-16c4c4673480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:01.851514 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.851482    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp" (OuterVolumeSpecName: "kube-api-access-v9fbp") pod "19581646-e9ee-48ee-9caf-16c4c4673480" (UID: "19581646-e9ee-48ee-9caf-16c4c4673480"). InnerVolumeSpecName "kube-api-access-v9fbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:01.950124 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.950071    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19581646-e9ee-48ee-9caf-16c4c4673480-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:01.950124 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:01.950118    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fbp\" (UniqueName: \"kubernetes.io/projected/19581646-e9ee-48ee-9caf-16c4c4673480-kube-api-access-v9fbp\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:02.454094 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.454060    2562 generic.go:356] "Generic (PLEG): container finished" podID="19581646-e9ee-48ee-9caf-16c4c4673480" containerID="f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2" exitCode=0
Sep 10 21:13:02.454314 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.454104    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"
Sep 10 21:13:02.454314 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.454132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" event={"ID":"19581646-e9ee-48ee-9caf-16c4c4673480","Type":"ContainerDied","Data":"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"}
Sep 10 21:13:02.454314 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.454206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7" event={"ID":"19581646-e9ee-48ee-9caf-16c4c4673480","Type":"ContainerDied","Data":"5a5023a6628dffd8b1b5e85473409a8bcdeb6627b23fbf4242d987f12a98d5d8"}
Sep 10 21:13:02.454314 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.454237    2562 scope.go:117] "RemoveContainer" containerID="f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"
Sep 10 21:13:02.462793 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.462774    2562 scope.go:117] "RemoveContainer" containerID="f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"
Sep 10 21:13:02.463106 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:13:02.463085    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2\": container with ID starting with f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2 not found: ID does not exist" containerID="f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"
Sep 10 21:13:02.463177 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.463115    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2"} err="failed to get container status \"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2\": rpc error: code = NotFound desc = could not find container \"f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2\": container with ID starting with f0ac35fe895574716e2cfa232211cfef5ce5294796750530f813b18dfab334c2 not found: ID does not exist"
Sep 10 21:13:02.474688 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.474658    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:13:02.477081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.477059    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-50ba47-768dbcf6f7-227v7"]
Sep 10 21:13:02.618481 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:02.618446    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19581646-e9ee-48ee-9caf-16c4c4673480" path="/var/lib/kubelet/pods/19581646-e9ee-48ee-9caf-16c4c4673480/volumes"
Sep 10 21:13:36.940764 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.940730    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:13:36.941234 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.941048    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19581646-e9ee-48ee-9caf-16c4c4673480" containerName="payload-processing"
Sep 10 21:13:36.941234 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.941059    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="19581646-e9ee-48ee-9caf-16c4c4673480" containerName="payload-processing"
Sep 10 21:13:36.941234 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.941127    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="19581646-e9ee-48ee-9caf-16c4c4673480" containerName="payload-processing"
Sep 10 21:13:36.944003 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.943986    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:36.946731 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.946706    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-766d8c20-dockercfg-xrnbv\""
Sep 10 21:13:36.946890 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.946775    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-766d8c20\""
Sep 10 21:13:36.953455 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:36.953433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:13:37.035715 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.035687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.035872 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.035773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5cmf\" (UniqueName: \"kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.136432 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.136387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5cmf\" (UniqueName: \"kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.136592 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.136450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.137166 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.137147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.150379 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.150348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5cmf\" (UniqueName: \"kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf\") pod \"payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") " pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.255193 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.255107    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.522216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.522187    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:13:37.524117 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:13:37.524089    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef6d178_4695_4165_bec7_7f1d5f4d0030.slice/crio-74c13305955503c6b0ba2f6b207dff66f0771e8282221b6c0668e16888c4b6aa WatchSource:0}: Error finding container 74c13305955503c6b0ba2f6b207dff66f0771e8282221b6c0668e16888c4b6aa: Status 404 returned error can't find the container with id 74c13305955503c6b0ba2f6b207dff66f0771e8282221b6c0668e16888c4b6aa
Sep 10 21:13:37.605168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.605136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" event={"ID":"1ef6d178-4695-4165-bec7-7f1d5f4d0030","Type":"ContainerStarted","Data":"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"}
Sep 10 21:13:37.605312 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.605193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" event={"ID":"1ef6d178-4695-4165-bec7-7f1d5f4d0030","Type":"ContainerStarted","Data":"74c13305955503c6b0ba2f6b207dff66f0771e8282221b6c0668e16888c4b6aa"}
Sep 10 21:13:37.605312 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.605215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:37.624390 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:37.624340    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" podStartSLOduration=1.62432045 podStartE2EDuration="1.62432045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:37.623732066 +0000 UTC m=+945.544352478" watchObservedRunningTime="2026-09-10 21:13:37.62432045 +0000 UTC m=+945.544940837"
Sep 10 21:13:45.498954 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:45.498919    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:13:48.621318 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:48.621289    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:13:58.164443 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.164405    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:13:58.164919 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.164676    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" podUID="9455174c-49e5-4f42-89fc-d2245cbd6e59" containerName="payload-processing" containerID="cri-o://bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122" gracePeriod=30
Sep 10 21:13:58.170892 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.170494    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:13:58.170892 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.170694    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" podUID="935d9dc6-0a54-4234-8249-f302f6c280f1" containerName="payload-pre-processing" containerID="cri-o://da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d" gracePeriod=30
Sep 10 21:13:58.442728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.442704    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:13:58.445359 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.445339    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:13:58.514518 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514487    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume\") pod \"935d9dc6-0a54-4234-8249-f302f6c280f1\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") "
Sep 10 21:13:58.514683 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsxmx\" (UniqueName: \"kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx\") pod \"935d9dc6-0a54-4234-8249-f302f6c280f1\" (UID: \"935d9dc6-0a54-4234-8249-f302f6c280f1\") "
Sep 10 21:13:58.514683 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514548    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume\") pod \"9455174c-49e5-4f42-89fc-d2245cbd6e59\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") "
Sep 10 21:13:58.514683 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514606    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnc4f\" (UniqueName: \"kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f\") pod \"9455174c-49e5-4f42-89fc-d2245cbd6e59\" (UID: \"9455174c-49e5-4f42-89fc-d2245cbd6e59\") "
Sep 10 21:13:58.514919 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "935d9dc6-0a54-4234-8249-f302f6c280f1" (UID: "935d9dc6-0a54-4234-8249-f302f6c280f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:58.514984 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.514941    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9455174c-49e5-4f42-89fc-d2245cbd6e59" (UID: "9455174c-49e5-4f42-89fc-d2245cbd6e59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:58.516708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.516681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx" (OuterVolumeSpecName: "kube-api-access-fsxmx") pod "935d9dc6-0a54-4234-8249-f302f6c280f1" (UID: "935d9dc6-0a54-4234-8249-f302f6c280f1"). InnerVolumeSpecName "kube-api-access-fsxmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:58.516774 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.516692    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f" (OuterVolumeSpecName: "kube-api-access-wnc4f") pod "9455174c-49e5-4f42-89fc-d2245cbd6e59" (UID: "9455174c-49e5-4f42-89fc-d2245cbd6e59"). InnerVolumeSpecName "kube-api-access-wnc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:58.615463 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.615433    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/935d9dc6-0a54-4234-8249-f302f6c280f1-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:58.615463 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.615458    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsxmx\" (UniqueName: \"kubernetes.io/projected/935d9dc6-0a54-4234-8249-f302f6c280f1-kube-api-access-fsxmx\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:58.615463 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.615468    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9455174c-49e5-4f42-89fc-d2245cbd6e59-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:58.615675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.615477    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc4f\" (UniqueName: \"kubernetes.io/projected/9455174c-49e5-4f42-89fc-d2245cbd6e59-kube-api-access-wnc4f\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:13:58.695707 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.695678    2562 generic.go:356] "Generic (PLEG): container finished" podID="9455174c-49e5-4f42-89fc-d2245cbd6e59" containerID="bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122" exitCode=0
Sep 10 21:13:58.695898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.695732    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"
Sep 10 21:13:58.695898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.695773    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" event={"ID":"9455174c-49e5-4f42-89fc-d2245cbd6e59","Type":"ContainerDied","Data":"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"}
Sep 10 21:13:58.695898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.695854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq" event={"ID":"9455174c-49e5-4f42-89fc-d2245cbd6e59","Type":"ContainerDied","Data":"3a3568230ba8a62963a233cb56bdf7453464a2b2e04e1ee08395c4cf767407a2"}
Sep 10 21:13:58.695898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.695885    2562 scope.go:117] "RemoveContainer" containerID="bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"
Sep 10 21:13:58.697071 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.697053    2562 generic.go:356] "Generic (PLEG): container finished" podID="935d9dc6-0a54-4234-8249-f302f6c280f1" containerID="da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d" exitCode=0
Sep 10 21:13:58.697151 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.697087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" event={"ID":"935d9dc6-0a54-4234-8249-f302f6c280f1","Type":"ContainerDied","Data":"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"}
Sep 10 21:13:58.697151 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.697129    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"
Sep 10 21:13:58.697231 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.697133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j" event={"ID":"935d9dc6-0a54-4234-8249-f302f6c280f1","Type":"ContainerDied","Data":"08676b769e001566f9a7ec1a0879295366bb8670316e10fbaf784044ec6562b2"}
Sep 10 21:13:58.704123 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.704105    2562 scope.go:117] "RemoveContainer" containerID="bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"
Sep 10 21:13:58.704358 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:13:58.704341    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122\": container with ID starting with bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122 not found: ID does not exist" containerID="bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"
Sep 10 21:13:58.704400 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.704365    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122"} err="failed to get container status \"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122\": rpc error: code = NotFound desc = could not find container \"bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122\": container with ID starting with bc2be9c5620c74e684bb4be6935c1f940948792935eb935fb5617285efc22122 not found: ID does not exist"
Sep 10 21:13:58.704400 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.704381    2562 scope.go:117] "RemoveContainer" containerID="da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"
Sep 10 21:13:58.711298 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.711279    2562 scope.go:117] "RemoveContainer" containerID="da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"
Sep 10 21:13:58.711535 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:13:58.711519    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d\": container with ID starting with da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d not found: ID does not exist" containerID="da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"
Sep 10 21:13:58.711576 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.711542    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d"} err="failed to get container status \"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d\": rpc error: code = NotFound desc = could not find container \"da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d\": container with ID starting with da930907fe52409e185ef9f472e08df51067c48e4887fe385a3da5f49358fb2d not found: ID does not exist"
Sep 10 21:13:58.714727 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.714708    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:13:58.717329 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.717311    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a41a24-7db5d59799-5xbmq"]
Sep 10 21:13:58.726984 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.726964    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:13:58.730587 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:13:58.730568    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a41a24-56659d7dfd-8dw8j"]
Sep 10 21:14:00.517949 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.517903    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" podUID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" containerName="main" containerID="cri-o://d2c59a338270ea72180ebd4b1bd4adb7bffa72ec4703be5caf4ba92f10ddb04f" gracePeriod=45
Sep 10 21:14:00.618235 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.618208    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935d9dc6-0a54-4234-8249-f302f6c280f1" path="/var/lib/kubelet/pods/935d9dc6-0a54-4234-8249-f302f6c280f1/volumes"
Sep 10 21:14:00.618571 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.618557    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9455174c-49e5-4f42-89fc-d2245cbd6e59" path="/var/lib/kubelet/pods/9455174c-49e5-4f42-89fc-d2245cbd6e59/volumes"
Sep 10 21:14:00.710298 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.710266    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" containerID="d2c59a338270ea72180ebd4b1bd4adb7bffa72ec4703be5caf4ba92f10ddb04f" exitCode=0
Sep 10 21:14:00.710440 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.710346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" event={"ID":"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0","Type":"ContainerDied","Data":"d2c59a338270ea72180ebd4b1bd4adb7bffa72ec4703be5caf4ba92f10ddb04f"}
Sep 10 21:14:00.759735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.759708    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:14:00.836943 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.836871    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.836943 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.836902    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.836943 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.836923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.837189 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.836961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.837189 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.836991    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f54h\" (UniqueName: \"kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.837189 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837034    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache\") pod \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\" (UID: \"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0\") "
Sep 10 21:14:00.837341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837188    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home" (OuterVolumeSpecName: "home") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:00.837341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:00.837434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837349    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:00.837434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837363    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:00.837434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.837366    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:00.839116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.839093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm" (OuterVolumeSpecName: "dshm") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:00.839199 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.839148    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:00.839280 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.839258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h" (OuterVolumeSpecName: "kube-api-access-8f54h") pod "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" (UID: "8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0"). InnerVolumeSpecName "kube-api-access-8f54h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:00.938493 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.938454    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:00.938493 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.938487    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:00.938682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.938503    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f54h\" (UniqueName: \"kubernetes.io/projected/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-kube-api-access-8f54h\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:00.938682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:00.938515    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.716951 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:01.716913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq" event={"ID":"8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0","Type":"ContainerDied","Data":"1ec10df4416fccd4127c7a445f71422f9658b2388192046a8691342eeb6903a2"}
Sep 10 21:14:01.716951 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:01.716939    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"
Sep 10 21:14:01.717444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:01.716970    2562 scope.go:117] "RemoveContainer" containerID="d2c59a338270ea72180ebd4b1bd4adb7bffa72ec4703be5caf4ba92f10ddb04f"
Sep 10 21:14:01.738896 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:01.738862    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:14:01.740180 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:01.740161    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a41a24/test-model-a41a24-kserve-745dfb5867-9jnlq"]
Sep 10 21:14:02.617664 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:02.617625    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" path="/var/lib/kubelet/pods/8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0/volumes"
Sep 10 21:14:07.472122 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472081    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"]
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472535    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9455174c-49e5-4f42-89fc-d2245cbd6e59" containerName="payload-processing"
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472553    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9455174c-49e5-4f42-89fc-d2245cbd6e59" containerName="payload-processing"
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472573    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935d9dc6-0a54-4234-8249-f302f6c280f1" containerName="payload-pre-processing"
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="935d9dc6-0a54-4234-8249-f302f6c280f1" containerName="payload-pre-processing"
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472593    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" containerName="main"
Sep 10 21:14:07.472609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472602    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" containerName="main"
Sep 10 21:14:07.472928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472687    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9455174c-49e5-4f42-89fc-d2245cbd6e59" containerName="payload-processing"
Sep 10 21:14:07.472928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472702    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ef4b3e0-05a3-4c62-a66c-eb938eed7ef0" containerName="main"
Sep 10 21:14:07.472928 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.472710    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="935d9dc6-0a54-4234-8249-f302f6c280f1" containerName="payload-pre-processing"
Sep 10 21:14:07.481249 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.481226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:07.485201 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.485174    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:14:07.485342 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.485174    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:14:07.486076 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.486049    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"]
Sep 10 21:14:07.595734 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.595704    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-766d8c20-d95mf\" (UID: \"39c100fc-0626-46cd-a8ce-d3de95e2671a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:07.697228 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.697188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-766d8c20-d95mf\" (UID: \"39c100fc-0626-46cd-a8ce-d3de95e2671a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:07.697888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.697862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-766d8c20-d95mf\" (UID: \"39c100fc-0626-46cd-a8ce-d3de95e2671a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:07.793324 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:07.793237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:08.058346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:08.058322    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"]
Sep 10 21:14:08.060733 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:14:08.060706    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c100fc_0626_46cd_a8ce_d3de95e2671a.slice/crio-d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a WatchSource:0}: Error finding container d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a: Status 404 returned error can't find the container with id d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a
Sep 10 21:14:08.062640 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:08.062621    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:14:08.748457 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:08.748421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf" event={"ID":"39c100fc-0626-46cd-a8ce-d3de95e2671a","Type":"ContainerStarted","Data":"d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a"}
Sep 10 21:14:10.759190 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:10.759158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf" event={"ID":"39c100fc-0626-46cd-a8ce-d3de95e2671a","Type":"ContainerStarted","Data":"de90c2ab3f44627ead654c626f5043816cef076ad1fc31a61e5ee96884338d88"}
Sep 10 21:14:10.773782 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:10.773727    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf" podStartSLOduration=1.496251163 podStartE2EDuration="3.773709908s" totalImagesPullingTime="2.277458745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:10.77300111 +0000 UTC m=+978.693621508" watchObservedRunningTime="2026-09-10 21:14:10.773709908 +0000 UTC m=+978.694330294"
Sep 10 21:14:11.765291 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:11.765258    2562 generic.go:356] "Generic (PLEG): container finished" podID="39c100fc-0626-46cd-a8ce-d3de95e2671a" containerID="de90c2ab3f44627ead654c626f5043816cef076ad1fc31a61e5ee96884338d88" exitCode=0
Sep 10 21:14:11.765651 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:11.765316    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf" event={"ID":"39c100fc-0626-46cd-a8ce-d3de95e2671a","Type":"ContainerDied","Data":"de90c2ab3f44627ead654c626f5043816cef076ad1fc31a61e5ee96884338d88"}
Sep 10 21:14:12.899824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:12.899779    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:13.045793 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.045712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca\") pod \"39c100fc-0626-46cd-a8ce-d3de95e2671a\" (UID: \"39c100fc-0626-46cd-a8ce-d3de95e2671a\") "
Sep 10 21:14:13.046087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.046062    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39c100fc-0626-46cd-a8ce-d3de95e2671a" (UID: "39c100fc-0626-46cd-a8ce-d3de95e2671a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:13.146797 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.146767    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c100fc-0626-46cd-a8ce-d3de95e2671a-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:13.774483 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.774453    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"
Sep 10 21:14:13.774643 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.774489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf" event={"ID":"39c100fc-0626-46cd-a8ce-d3de95e2671a","Type":"ContainerDied","Data":"d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a"}
Sep 10 21:14:13.774643 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:13.774534    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d048ef9881e92f4f95b6e7dc8dd0584301c9b251861eebf5eb10e4e7be3bb28a"
Sep 10 21:14:16.357228 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.357147    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:14:16.357679 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.357394    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" podUID="fd993c51-32c7-44fc-8954-18a4017cd94c" containerName="payload-processing" containerID="cri-o://3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e" gracePeriod=30
Sep 10 21:14:16.593537 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.593513    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:14:16.777655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.777618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume\") pod \"fd993c51-32c7-44fc-8954-18a4017cd94c\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") "
Sep 10 21:14:16.777876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.777715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkpg4\" (UniqueName: \"kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4\") pod \"fd993c51-32c7-44fc-8954-18a4017cd94c\" (UID: \"fd993c51-32c7-44fc-8954-18a4017cd94c\") "
Sep 10 21:14:16.778053 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.778027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd993c51-32c7-44fc-8954-18a4017cd94c" (UID: "fd993c51-32c7-44fc-8954-18a4017cd94c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:16.779852 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.779826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4" (OuterVolumeSpecName: "kube-api-access-hkpg4") pod "fd993c51-32c7-44fc-8954-18a4017cd94c" (UID: "fd993c51-32c7-44fc-8954-18a4017cd94c"). InnerVolumeSpecName "kube-api-access-hkpg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:16.788762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.788735    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd993c51-32c7-44fc-8954-18a4017cd94c" containerID="3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e" exitCode=0
Sep 10 21:14:16.788877 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.788773    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"
Sep 10 21:14:16.788877 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.788829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" event={"ID":"fd993c51-32c7-44fc-8954-18a4017cd94c","Type":"ContainerDied","Data":"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"}
Sep 10 21:14:16.788950 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.788888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn" event={"ID":"fd993c51-32c7-44fc-8954-18a4017cd94c","Type":"ContainerDied","Data":"e892e8c77b885b875e4c619ce495c725568522ebddc383a77473762e32cbe93b"}
Sep 10 21:14:16.788950 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.788912    2562 scope.go:117] "RemoveContainer" containerID="3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"
Sep 10 21:14:16.797285 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.797266    2562 scope.go:117] "RemoveContainer" containerID="3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"
Sep 10 21:14:16.797545 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:14:16.797524    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e\": container with ID starting with 3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e not found: ID does not exist" containerID="3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"
Sep 10 21:14:16.797601 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.797552    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e"} err="failed to get container status \"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e\": rpc error: code = NotFound desc = could not find container \"3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e\": container with ID starting with 3ae37f7a597324d7f2ef5d6079222d5657f083539ee9323cb4d9f93167ef777e not found: ID does not exist"
Sep 10 21:14:16.808956 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.808933    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:14:16.817907 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.817885    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69dcc8-f54447965-9skgn"]
Sep 10 21:14:16.878736 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.878699    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkpg4\" (UniqueName: \"kubernetes.io/projected/fd993c51-32c7-44fc-8954-18a4017cd94c-kube-api-access-hkpg4\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:16.878736 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:16.878735    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd993c51-32c7-44fc-8954-18a4017cd94c-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:18.264989 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.264949    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:14:18.265406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.265190    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" podUID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" containerName="payload-processing" containerID="cri-o://31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5" gracePeriod=30
Sep 10 21:14:18.421778 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.421742    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"]
Sep 10 21:14:18.425719 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.425687    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-766d8c20-d95mf"]
Sep 10 21:14:18.524344 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.524290    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:14:18.617721 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.617691    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c100fc-0626-46cd-a8ce-d3de95e2671a" path="/var/lib/kubelet/pods/39c100fc-0626-46cd-a8ce-d3de95e2671a/volumes"
Sep 10 21:14:18.618090 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.618075    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd993c51-32c7-44fc-8954-18a4017cd94c" path="/var/lib/kubelet/pods/fd993c51-32c7-44fc-8954-18a4017cd94c/volumes"
Sep 10 21:14:18.698331 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.698301    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5cmf\" (UniqueName: \"kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf\") pod \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") "
Sep 10 21:14:18.698468 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.698348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume\") pod \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\" (UID: \"1ef6d178-4695-4165-bec7-7f1d5f4d0030\") "
Sep 10 21:14:18.698720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.698697    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ef6d178-4695-4165-bec7-7f1d5f4d0030" (UID: "1ef6d178-4695-4165-bec7-7f1d5f4d0030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:18.700329 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.700308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf" (OuterVolumeSpecName: "kube-api-access-p5cmf") pod "1ef6d178-4695-4165-bec7-7f1d5f4d0030" (UID: "1ef6d178-4695-4165-bec7-7f1d5f4d0030"). InnerVolumeSpecName "kube-api-access-p5cmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:18.799137 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799048    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5cmf\" (UniqueName: \"kubernetes.io/projected/1ef6d178-4695-4165-bec7-7f1d5f4d0030-kube-api-access-p5cmf\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:18.799137 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799116    2562 generic.go:356] "Generic (PLEG): container finished" podID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" containerID="31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5" exitCode=0
Sep 10 21:14:18.799354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799163    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef6d178-4695-4165-bec7-7f1d5f4d0030-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:14:18.799354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" event={"ID":"1ef6d178-4695-4165-bec7-7f1d5f4d0030","Type":"ContainerDied","Data":"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"}
Sep 10 21:14:18.799354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799240    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"
Sep 10 21:14:18.799354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv" event={"ID":"1ef6d178-4695-4165-bec7-7f1d5f4d0030","Type":"ContainerDied","Data":"74c13305955503c6b0ba2f6b207dff66f0771e8282221b6c0668e16888c4b6aa"}
Sep 10 21:14:18.799354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.799319    2562 scope.go:117] "RemoveContainer" containerID="31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"
Sep 10 21:14:18.807696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.807680    2562 scope.go:117] "RemoveContainer" containerID="31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"
Sep 10 21:14:18.807969 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:14:18.807950    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5\": container with ID starting with 31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5 not found: ID does not exist" containerID="31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"
Sep 10 21:14:18.808012 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.807977    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5"} err="failed to get container status \"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5\": rpc error: code = NotFound desc = could not find container \"31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5\": container with ID starting with 31e4d51f72b83f463ebdff519b48fc691bf3b09436068d22ffe5c1db568b8db5 not found: ID does not exist"
Sep 10 21:14:18.819775 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.819755    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:14:18.825634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:18.825613    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-766d8c20-7fb5bf5d9-7cgtv"]
Sep 10 21:14:20.618731 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:20.618697    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" path="/var/lib/kubelet/pods/1ef6d178-4695-4165-bec7-7f1d5f4d0030/volumes"
Sep 10 21:14:28.463993 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.463952    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464245    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c100fc-0626-46cd-a8ce-d3de95e2671a" containerName="revoke-keys"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464256    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c100fc-0626-46cd-a8ce-d3de95e2671a" containerName="revoke-keys"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464270    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd993c51-32c7-44fc-8954-18a4017cd94c" containerName="payload-processing"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464276    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd993c51-32c7-44fc-8954-18a4017cd94c" containerName="payload-processing"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464284    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" containerName="payload-processing"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464290    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" containerName="payload-processing"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464340    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef6d178-4695-4165-bec7-7f1d5f4d0030" containerName="payload-processing"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464348    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c100fc-0626-46cd-a8ce-d3de95e2671a" containerName="revoke-keys"
Sep 10 21:14:28.464418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.464357    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd993c51-32c7-44fc-8954-18a4017cd94c" containerName="payload-processing"
Sep 10 21:14:28.467090 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.467074    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.469855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.469834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8623b2-dockercfg-qwv7b\""
Sep 10 21:14:28.469974 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.469905    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8623b2\""
Sep 10 21:14:28.478864 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.478842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:14:28.538406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.538378    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:14:28.541564 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.541545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.564632 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.564610    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:14:28.566374 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.566350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xrz\" (UniqueName: \"kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.566458 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.566394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.566509 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.566471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnm7m\" (UniqueName: \"kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.566556 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.566504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.666996 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.666966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.667146 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.667061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnm7m\" (UniqueName: \"kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.667146 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.667093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.667146 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.667142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xrz\" (UniqueName: \"kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.667907 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.667886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.667961 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.667921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.675312 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.675282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnm7m\" (UniqueName: \"kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m\") pod \"payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.675405 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.675282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xrz\" (UniqueName: \"kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz\") pod \"payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:28.776866 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.776792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:28.850301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:28.850275    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:29.049540 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.049511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:14:29.051461 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:14:29.051435    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d84bfd5_3699_41ca_81cd_d241b2e9b0e0.slice/crio-c8f09611130e3aa772846603a0b0f6de1251b044a95dbb46a52918f3ef3db9ba WatchSource:0}: Error finding container c8f09611130e3aa772846603a0b0f6de1251b044a95dbb46a52918f3ef3db9ba: Status 404 returned error can't find the container with id c8f09611130e3aa772846603a0b0f6de1251b044a95dbb46a52918f3ef3db9ba
Sep 10 21:14:29.124265 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.124243    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:14:29.126042 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:14:29.126017    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb651fb2f_5918_4139_81b6_af25d3c0c151.slice/crio-203fa7c61e81978f7a9503b15dbda1046397c8204c4eb40d54b5be376bd7231b WatchSource:0}: Error finding container 203fa7c61e81978f7a9503b15dbda1046397c8204c4eb40d54b5be376bd7231b: Status 404 returned error can't find the container with id 203fa7c61e81978f7a9503b15dbda1046397c8204c4eb40d54b5be376bd7231b
Sep 10 21:14:29.849689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.849654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" event={"ID":"b651fb2f-5918-4139-81b6-af25d3c0c151","Type":"ContainerStarted","Data":"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"}
Sep 10 21:14:29.850177 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.849719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" event={"ID":"b651fb2f-5918-4139-81b6-af25d3c0c151","Type":"ContainerStarted","Data":"203fa7c61e81978f7a9503b15dbda1046397c8204c4eb40d54b5be376bd7231b"}
Sep 10 21:14:29.850177 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.849904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:29.851306 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.851282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" event={"ID":"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0","Type":"ContainerStarted","Data":"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"}
Sep 10 21:14:29.851440 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.851320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" event={"ID":"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0","Type":"ContainerStarted","Data":"c8f09611130e3aa772846603a0b0f6de1251b044a95dbb46a52918f3ef3db9ba"}
Sep 10 21:14:29.851440 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.851347    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:29.868398 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.868337    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" podStartSLOduration=1.868318001 podStartE2EDuration="1.868318001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:29.866607029 +0000 UTC m=+997.787227441" watchObservedRunningTime="2026-09-10 21:14:29.868318001 +0000 UTC m=+997.788938392"
Sep 10 21:14:29.883586 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:29.883538    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" podStartSLOduration=1.883525675 podStartE2EDuration="1.883525675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:29.881684781 +0000 UTC m=+997.802305182" watchObservedRunningTime="2026-09-10 21:14:29.883525675 +0000 UTC m=+997.804146060"
Sep 10 21:14:40.858601 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:40.858569    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:14:40.859104 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:40.858642    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:14:43.515996 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.515960    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:14:43.519508 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.519483    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.522070 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.522044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-91c25e-dockercfg-4wn9s\""
Sep 10 21:14:43.522206 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.522116    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-91c25e\""
Sep 10 21:14:43.528883 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.528860    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:14:43.591531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.591504    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkr59\" (UniqueName: \"kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.591669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.591544    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.692555 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.692517    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkr59\" (UniqueName: \"kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.692725 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.692563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.693269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.693251    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.700704 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.700682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkr59\" (UniqueName: \"kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59\") pod \"payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:43.831515 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:43.831430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:44.096433 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:44.096407    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:14:44.098390 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:14:44.098361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4c0db9_85d9_4f1e_b2a8_1349b0064363.slice/crio-94db9676610a42f5df0b09ff2f69f79670026ee91cbe29efc0129417e6b29c86 WatchSource:0}: Error finding container 94db9676610a42f5df0b09ff2f69f79670026ee91cbe29efc0129417e6b29c86: Status 404 returned error can't find the container with id 94db9676610a42f5df0b09ff2f69f79670026ee91cbe29efc0129417e6b29c86
Sep 10 21:14:44.921216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:44.921173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" event={"ID":"8f4c0db9-85d9-4f1e-b2a8-1349b0064363","Type":"ContainerStarted","Data":"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"}
Sep 10 21:14:44.921581 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:44.921243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" event={"ID":"8f4c0db9-85d9-4f1e-b2a8-1349b0064363","Type":"ContainerStarted","Data":"94db9676610a42f5df0b09ff2f69f79670026ee91cbe29efc0129417e6b29c86"}
Sep 10 21:14:44.921581 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:44.921276    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:44.938404 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:44.938353    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" podStartSLOduration=1.938339382 podStartE2EDuration="1.938339382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:44.93625377 +0000 UTC m=+1012.856874155" watchObservedRunningTime="2026-09-10 21:14:44.938339382 +0000 UTC m=+1012.858959767"
Sep 10 21:14:55.933541 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:55.928148    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:14:56.410355 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.410321    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:14:56.414997 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.414976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.417746 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.417721    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4eb314\""
Sep 10 21:14:56.417883 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.417797    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4eb314-dockercfg-ws45z\""
Sep 10 21:14:56.426814 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.426779    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:14:56.490013 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.489985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9zq\" (UniqueName: \"kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.490135 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.490025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.591287 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.591255    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9zq\" (UniqueName: \"kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.591413 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.591298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.591977 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.591959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.600018 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.599996    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9zq\" (UniqueName: \"kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq\") pod \"payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.724361 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.724335    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:56.991097 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:56.991073    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:14:56.992758 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:14:56.992730    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42783ce_3f4f_40e1_91d2_3e948ca98314.slice/crio-f24c1b2bfa8678c582916359c44c559a68beabefa49001dcff40ccbda4900474 WatchSource:0}: Error finding container f24c1b2bfa8678c582916359c44c559a68beabefa49001dcff40ccbda4900474: Status 404 returned error can't find the container with id f24c1b2bfa8678c582916359c44c559a68beabefa49001dcff40ccbda4900474
Sep 10 21:14:57.981170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:57.981125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" event={"ID":"c42783ce-3f4f-40e1-91d2-3e948ca98314","Type":"ContainerStarted","Data":"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"}
Sep 10 21:14:57.981380 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:57.981200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" event={"ID":"c42783ce-3f4f-40e1-91d2-3e948ca98314","Type":"ContainerStarted","Data":"f24c1b2bfa8678c582916359c44c559a68beabefa49001dcff40ccbda4900474"}
Sep 10 21:14:57.981380 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:57.981233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:14:57.995961 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:14:57.995917    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" podStartSLOduration=1.995903411 podStartE2EDuration="1.995903411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:57.9954924 +0000 UTC m=+1025.916112807" watchObservedRunningTime="2026-09-10 21:14:57.995903411 +0000 UTC m=+1025.916523797"
Sep 10 21:15:00.137005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.136969    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"]
Sep 10 21:15:00.140738 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.140717    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"]
Sep 10 21:15:00.140876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.140840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"
Sep 10 21:15:00.143387 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.143365    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:15:00.143791 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.143774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"
Sep 10 21:15:00.150713 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.150577    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"]
Sep 10 21:15:00.152681 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.152658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"]
Sep 10 21:15:00.152867 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.152851    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"
Sep 10 21:15:00.160071 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.160056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"
Sep 10 21:15:00.431081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.431005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"]
Sep 10 21:15:00.435208 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.435185    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"]
Sep 10 21:15:00.435597 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:00.435572    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb11260_d76f_4e08_8354_06bd58501f62.slice/crio-7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63 WatchSource:0}: Error finding container 7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63: Status 404 returned error can't find the container with id 7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63
Sep 10 21:15:00.439516 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:00.439482    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2296c82b_a27d_413a_8fed_2d18c209adfe.slice/crio-f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15 WatchSource:0}: Error finding container f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15: Status 404 returned error can't find the container with id f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15
Sep 10 21:15:00.995456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.995414    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7" event={"ID":"fcb11260-d76f-4e08-8354-06bd58501f62","Type":"ContainerStarted","Data":"3c460d0d6a16f31824ca5aef5b04f2d0165ba7a9d5142482966871e0f4aaaf26"}
Sep 10 21:15:00.995613 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.995481    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7" event={"ID":"fcb11260-d76f-4e08-8354-06bd58501f62","Type":"ContainerStarted","Data":"7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63"}
Sep 10 21:15:00.996935 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.996913    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs" event={"ID":"2296c82b-a27d-413a-8fed-2d18c209adfe","Type":"ContainerStarted","Data":"16b6c5c4f3ef4eb20617e8be94ab6582aacdde056c7fae4f2aa72f579a717479"}
Sep 10 21:15:00.997030 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:00.996950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs" event={"ID":"2296c82b-a27d-413a-8fed-2d18c209adfe","Type":"ContainerStarted","Data":"f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15"}
Sep 10 21:15:01.011038 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:01.010964    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7" podStartSLOduration=1.010952635 podStartE2EDuration="1.010952635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:01.010117845 +0000 UTC m=+1028.930738228" watchObservedRunningTime="2026-09-10 21:15:01.010952635 +0000 UTC m=+1028.931573042"
Sep 10 21:15:01.024263 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:01.024217    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs" podStartSLOduration=1.024204003 podStartE2EDuration="1.024204003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:01.023917872 +0000 UTC m=+1028.944538277" watchObservedRunningTime="2026-09-10 21:15:01.024204003 +0000 UTC m=+1028.944824389"
Sep 10 21:15:02.002823 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:02.002780    2562 generic.go:356] "Generic (PLEG): container finished" podID="2296c82b-a27d-413a-8fed-2d18c209adfe" containerID="16b6c5c4f3ef4eb20617e8be94ab6582aacdde056c7fae4f2aa72f579a717479" exitCode=0
Sep 10 21:15:02.003197 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:02.002861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs" event={"ID":"2296c82b-a27d-413a-8fed-2d18c209adfe","Type":"ContainerDied","Data":"16b6c5c4f3ef4eb20617e8be94ab6582aacdde056c7fae4f2aa72f579a717479"}
Sep 10 21:15:02.004346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:02.004325    2562 generic.go:356] "Generic (PLEG): container finished" podID="fcb11260-d76f-4e08-8354-06bd58501f62" containerID="3c460d0d6a16f31824ca5aef5b04f2d0165ba7a9d5142482966871e0f4aaaf26" exitCode=0
Sep 10 21:15:02.004447 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:02.004356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7" event={"ID":"fcb11260-d76f-4e08-8354-06bd58501f62","Type":"ContainerDied","Data":"3c460d0d6a16f31824ca5aef5b04f2d0165ba7a9d5142482966871e0f4aaaf26"}
Sep 10 21:15:03.163339 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:03.163246    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"
Sep 10 21:15:03.165934 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:03.165913    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"
Sep 10 21:15:04.014469 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.014443    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"
Sep 10 21:15:04.014649 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.014484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs" event={"ID":"2296c82b-a27d-413a-8fed-2d18c209adfe","Type":"ContainerDied","Data":"f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15"}
Sep 10 21:15:04.014649 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.014536    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3116fc03f96c5638299793c46c658993e33e1ecd7d2fb7eb406713f533a7d15"
Sep 10 21:15:04.015822 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.015780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7" event={"ID":"fcb11260-d76f-4e08-8354-06bd58501f62","Type":"ContainerDied","Data":"7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63"}
Sep 10 21:15:04.015955 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.015836    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7105f52c67f8173ab280b01e27b7074adc130df7898702ff1e588aef12672c63"
Sep 10 21:15:04.015955 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:04.015786    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-vr9g7"
Sep 10 21:15:08.987499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:08.987471    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:15:12.486425 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.486384    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:15:12.487034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.486885    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2296c82b-a27d-413a-8fed-2d18c209adfe" containerName="cleanup"
Sep 10 21:15:12.487034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.486904    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2296c82b-a27d-413a-8fed-2d18c209adfe" containerName="cleanup"
Sep 10 21:15:12.487034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.486933    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb11260-d76f-4e08-8354-06bd58501f62" containerName="cleanup"
Sep 10 21:15:12.487034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.486942    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb11260-d76f-4e08-8354-06bd58501f62" containerName="cleanup"
Sep 10 21:15:12.487034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.487032    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2296c82b-a27d-413a-8fed-2d18c209adfe" containerName="cleanup"
Sep 10 21:15:12.487297 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.487044    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb11260-d76f-4e08-8354-06bd58501f62" containerName="cleanup"
Sep 10 21:15:12.489846 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.489826    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.495274 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.495255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-837e16-dockercfg-t76dr\""
Sep 10 21:15:12.496203 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.496136    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-837e16\""
Sep 10 21:15:12.503718 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.503698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:15:12.622029 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.621998    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.622178 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.622100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qgt\" (UniqueName: \"kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.723082 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.723053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.723223 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.723196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qgt\" (UniqueName: \"kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.723709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.723691    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.730851 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.730827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qgt\" (UniqueName: \"kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt\") pod \"payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:12.799689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:12.799621    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:13.066358 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:13.066334    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:15:13.068426 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:13.068396    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2f3442_70d7_413d_b180_c4dbf0a97f0d.slice/crio-a241ea8e70ac655b3890630572d377c4fa2d3bfcbf3ba5738083d4b9eabe88e9 WatchSource:0}: Error finding container a241ea8e70ac655b3890630572d377c4fa2d3bfcbf3ba5738083d4b9eabe88e9: Status 404 returned error can't find the container with id a241ea8e70ac655b3890630572d377c4fa2d3bfcbf3ba5738083d4b9eabe88e9
Sep 10 21:15:14.065334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:14.065300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" event={"ID":"3f2f3442-70d7-413d-b180-c4dbf0a97f0d","Type":"ContainerStarted","Data":"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"}
Sep 10 21:15:14.065730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:14.065352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" event={"ID":"3f2f3442-70d7-413d-b180-c4dbf0a97f0d","Type":"ContainerStarted","Data":"a241ea8e70ac655b3890630572d377c4fa2d3bfcbf3ba5738083d4b9eabe88e9"}
Sep 10 21:15:14.065730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:14.065390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:14.081078 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:14.081034    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" podStartSLOduration=2.081017839 podStartE2EDuration="2.081017839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:14.080338077 +0000 UTC m=+1042.000958478" watchObservedRunningTime="2026-09-10 21:15:14.081017839 +0000 UTC m=+1042.001638257"
Sep 10 21:15:25.072237 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:25.072206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:15:36.399202 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.399169    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7734-ac8722"]
Sep 10 21:15:36.404098 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.404072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:36.405541 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.405515    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-ac8722"]
Sep 10 21:15:36.406492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.406468    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8md8h\""
Sep 10 21:15:36.417774 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.417749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7np\" (UniqueName: \"kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np\") pod \"test-curl-7734-ac8722\" (UID: \"6f535429-538e-4a75-a96c-fd543d26baab\") " pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:36.518901 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.518875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7np\" (UniqueName: \"kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np\") pod \"test-curl-7734-ac8722\" (UID: \"6f535429-538e-4a75-a96c-fd543d26baab\") " pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:36.526595 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.526562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7np\" (UniqueName: \"kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np\") pod \"test-curl-7734-ac8722\" (UID: \"6f535429-538e-4a75-a96c-fd543d26baab\") " pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:36.713942 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.713916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:36.980867 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:36.980841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-ac8722"]
Sep 10 21:15:36.983031 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:36.983008    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f535429_538e_4a75_a96c_fd543d26baab.slice/crio-283abcb41001c90b1e1540500d09bfa53fe32293616fc0b96b54d2ae2d208eef WatchSource:0}: Error finding container 283abcb41001c90b1e1540500d09bfa53fe32293616fc0b96b54d2ae2d208eef: Status 404 returned error can't find the container with id 283abcb41001c90b1e1540500d09bfa53fe32293616fc0b96b54d2ae2d208eef
Sep 10 21:15:37.179216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.179182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-ac8722" event={"ID":"6f535429-538e-4a75-a96c-fd543d26baab","Type":"ContainerStarted","Data":"3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac"}
Sep 10 21:15:37.179385 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.179227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-ac8722" event={"ID":"6f535429-538e-4a75-a96c-fd543d26baab","Type":"ContainerStarted","Data":"283abcb41001c90b1e1540500d09bfa53fe32293616fc0b96b54d2ae2d208eef"}
Sep 10 21:15:37.193153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.193105    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7734-ac8722" podStartSLOduration=1.193091542 podStartE2EDuration="1.193091542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:37.192190175 +0000 UTC m=+1065.112810561" watchObservedRunningTime="2026-09-10 21:15:37.193091542 +0000 UTC m=+1065.113711927"
Sep 10 21:15:37.766658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.766622    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7734-ac8722"]
Sep 10 21:15:37.768594 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.768568    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7734-ac8722"]
Sep 10 21:15:37.880872 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.880846    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7734-c978c8"]
Sep 10 21:15:37.881179 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.881164    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f535429-538e-4a75-a96c-fd543d26baab" containerName="test-curl-7734-ac8722"
Sep 10 21:15:37.881224 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.881182    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f535429-538e-4a75-a96c-fd543d26baab" containerName="test-curl-7734-ac8722"
Sep 10 21:15:37.881261 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.881240    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f535429-538e-4a75-a96c-fd543d26baab" containerName="test-curl-7734-ac8722"
Sep 10 21:15:37.884302 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.884287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:37.888030 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.888004    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-c978c8"]
Sep 10 21:15:37.929902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:37.929882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nnm\" (UniqueName: \"kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm\") pod \"test-curl-7734-c978c8\" (UID: \"2d4e1105-80b5-46ba-b3e9-3b8913265605\") " pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:38.030741 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:38.030674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nnm\" (UniqueName: \"kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm\") pod \"test-curl-7734-c978c8\" (UID: \"2d4e1105-80b5-46ba-b3e9-3b8913265605\") " pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:38.038763 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:38.038743    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nnm\" (UniqueName: \"kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm\") pod \"test-curl-7734-c978c8\" (UID: \"2d4e1105-80b5-46ba-b3e9-3b8913265605\") " pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:38.183556 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:38.183533    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7734-ac8722" container="test-curl-7734-ac8722"
Sep 10 21:15:38.194281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:38.194262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:38.459565 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:38.459533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-c978c8"]
Sep 10 21:15:39.190222 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.190191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-c978c8" event={"ID":"2d4e1105-80b5-46ba-b3e9-3b8913265605","Type":"ContainerStarted","Data":"3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8"}
Sep 10 21:15:39.190222 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.190231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-c978c8" event={"ID":"2d4e1105-80b5-46ba-b3e9-3b8913265605","Type":"ContainerStarted","Data":"cddb666370632bc7c69b4c96bbbf4d600aa4f0aed9d6b13f6fa56bde2e52886d"}
Sep 10 21:15:39.190661 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.190327    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7734-ac8722" podUID="6f535429-538e-4a75-a96c-fd543d26baab" containerName="test-curl-7734-ac8722" containerID="cri-o://3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac" gracePeriod=2
Sep 10 21:15:39.204536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.204494    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7734-c978c8" podStartSLOduration=2.204479005 podStartE2EDuration="2.204479005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:39.203147506 +0000 UTC m=+1067.123767893" watchObservedRunningTime="2026-09-10 21:15:39.204479005 +0000 UTC m=+1067.125099390"
Sep 10 21:15:39.550675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.550602    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:15:39.553958 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.553937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.556539 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.556519    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"openshift-service-ca.crt\""
Sep 10 21:15:39.557898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.557875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"auth-test-model-8623b2-kserve-self-signed-certs\""
Sep 10 21:15:39.558007 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.557914    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"kube-root-ca.crt\""
Sep 10 21:15:39.558072 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.558032    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"default-dockercfg-t2sp2\""
Sep 10 21:15:39.565714 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.565694    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:15:39.643401 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.643544 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.643544 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.643544 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643526    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.643715 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278h2\" (UniqueName: \"kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.643715 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.643617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.744699 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.744873 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278h2\" (UniqueName: \"kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.744873 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.744873 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.744873 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.745112 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.744900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.745112 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.745054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.745223 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.745157    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.745279 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.745236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.747167 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.747146    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.747405 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.747390    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.751867 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.751844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-278h2\" (UniqueName: \"kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2\") pod \"auth-test-model-8623b2-kserve-7887b7cc9-hlnwp\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.777192 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.777170    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7734-c978c8"]
Sep 10 21:15:39.778971 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.778951    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7734-c978c8"]
Sep 10 21:15:39.866602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.866529    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:39.891923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.891895    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7734-4d39b5"]
Sep 10 21:15:39.892241 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.892223    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" containerName="test-curl-7734-c978c8"
Sep 10 21:15:39.892241 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.892239    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" containerName="test-curl-7734-c978c8"
Sep 10 21:15:39.892380 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.892314    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" containerName="test-curl-7734-c978c8"
Sep 10 21:15:39.895341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.895321    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:39.901073 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.901050    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-4d39b5"]
Sep 10 21:15:39.947239 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:39.947206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlw8l\" (UniqueName: \"kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l\") pod \"test-curl-7734-4d39b5\" (UID: \"7580acc5-269f-423a-98ba-cc87f505c40d\") " pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:40.048584 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.048560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlw8l\" (UniqueName: \"kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l\") pod \"test-curl-7734-4d39b5\" (UID: \"7580acc5-269f-423a-98ba-cc87f505c40d\") " pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:40.056299 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.056273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlw8l\" (UniqueName: \"kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l\") pod \"test-curl-7734-4d39b5\" (UID: \"7580acc5-269f-423a-98ba-cc87f505c40d\") " pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:40.143765 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.143742    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:15:40.145402 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:40.145378    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e3f345_4e63_4f6c_9281_14e40d9bd1c8.slice/crio-873b00f00a971b1708d5ae49a6e6dacee90895d37f5c249594704e88dff564d1 WatchSource:0}: Error finding container 873b00f00a971b1708d5ae49a6e6dacee90895d37f5c249594704e88dff564d1: Status 404 returned error can't find the container with id 873b00f00a971b1708d5ae49a6e6dacee90895d37f5c249594704e88dff564d1
Sep 10 21:15:40.197717 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.197695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" event={"ID":"12e3f345-4e63-4f6c-9281-14e40d9bd1c8","Type":"ContainerStarted","Data":"873b00f00a971b1708d5ae49a6e6dacee90895d37f5c249594704e88dff564d1"}
Sep 10 21:15:40.198000 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:40.197933    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7734-c978c8" container="test-curl-7734-c978c8"
Sep 10 21:15:40.206383 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.206357    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:40.472994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:40.472970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-4d39b5"]
Sep 10 21:15:40.474682 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:40.474655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7580acc5_269f_423a_98ba_cc87f505c40d.slice/crio-18d64efb7410792d9f0e2527ed3d48adc4a0761b4b629be836f1eaabe468f2bb WatchSource:0}: Error finding container 18d64efb7410792d9f0e2527ed3d48adc4a0761b4b629be836f1eaabe468f2bb: Status 404 returned error can't find the container with id 18d64efb7410792d9f0e2527ed3d48adc4a0761b4b629be836f1eaabe468f2bb
Sep 10 21:15:41.203837 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.203788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-4d39b5" event={"ID":"7580acc5-269f-423a-98ba-cc87f505c40d","Type":"ContainerStarted","Data":"0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e"}
Sep 10 21:15:41.204228 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.203856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-4d39b5" event={"ID":"7580acc5-269f-423a-98ba-cc87f505c40d","Type":"ContainerStarted","Data":"18d64efb7410792d9f0e2527ed3d48adc4a0761b4b629be836f1eaabe468f2bb"}
Sep 10 21:15:41.205534 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.205512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" event={"ID":"12e3f345-4e63-4f6c-9281-14e40d9bd1c8","Type":"ContainerStarted","Data":"120eb803f50ac25c255872f128ef3e9ab2b8d1509585be55de90990bf7b643d5"}
Sep 10 21:15:41.205652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.205634    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7734-c978c8" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" containerName="test-curl-7734-c978c8" containerID="cri-o://3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8" gracePeriod=2
Sep 10 21:15:41.219232 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.219194    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7734-4d39b5" podStartSLOduration=2.219180863 podStartE2EDuration="2.219180863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:41.217663232 +0000 UTC m=+1069.138283629" watchObservedRunningTime="2026-09-10 21:15:41.219180863 +0000 UTC m=+1069.139801250"
Sep 10 21:15:41.233225 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.233189    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" podStartSLOduration=2.233174088 podStartE2EDuration="2.233174088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:41.233014979 +0000 UTC m=+1069.153635368" watchObservedRunningTime="2026-09-10 21:15:41.233174088 +0000 UTC m=+1069.153794474"
Sep 10 21:15:41.353686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.353659    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:41.466139 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.466065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb7np\" (UniqueName: \"kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np\") pod \"6f535429-538e-4a75-a96c-fd543d26baab\" (UID: \"6f535429-538e-4a75-a96c-fd543d26baab\") "
Sep 10 21:15:41.467997 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.467971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np" (OuterVolumeSpecName: "kube-api-access-nb7np") pod "6f535429-538e-4a75-a96c-fd543d26baab" (UID: "6f535429-538e-4a75-a96c-fd543d26baab"). InnerVolumeSpecName "kube-api-access-nb7np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:41.566719 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.566691    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7np\" (UniqueName: \"kubernetes.io/projected/6f535429-538e-4a75-a96c-fd543d26baab-kube-api-access-nb7np\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:15:41.764156 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.764070    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7734-4d39b5"]
Sep 10 21:15:41.768024 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.767995    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7734-4d39b5"]
Sep 10 21:15:41.878107 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.878079    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7734-0411ef"]
Sep 10 21:15:41.878456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.878442    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" containerName="test-curl-7734-4d39b5"
Sep 10 21:15:41.878503 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.878458    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" containerName="test-curl-7734-4d39b5"
Sep 10 21:15:41.878538 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.878519    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" containerName="test-curl-7734-4d39b5"
Sep 10 21:15:41.881470 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.881456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:41.885047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.885023    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-0411ef"]
Sep 10 21:15:41.970149 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:41.970113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcg7g\" (UniqueName: \"kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g\") pod \"test-curl-7734-0411ef\" (UID: \"69d9cb73-4f57-4fad-a9f2-4f1037ebe586\") " pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:42.070999 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.070929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcg7g\" (UniqueName: \"kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g\") pod \"test-curl-7734-0411ef\" (UID: \"69d9cb73-4f57-4fad-a9f2-4f1037ebe586\") " pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:42.079045 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.079022    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcg7g\" (UniqueName: \"kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g\") pod \"test-curl-7734-0411ef\" (UID: \"69d9cb73-4f57-4fad-a9f2-4f1037ebe586\") " pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:42.191046 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.191015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:42.211978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.211953    2562 generic.go:356] "Generic (PLEG): container finished" podID="6f535429-538e-4a75-a96c-fd543d26baab" containerID="3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac" exitCode=137
Sep 10 21:15:42.212415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.211999    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:42.212415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.212050    2562 scope.go:117] "RemoveContainer" containerID="3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac"
Sep 10 21:15:42.212415 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:42.212297    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7734-4d39b5" container="test-curl-7734-4d39b5"
Sep 10 21:15:42.221338 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.221321    2562 scope.go:117] "RemoveContainer" containerID="3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac"
Sep 10 21:15:42.221621 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:42.221594    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac\": container with ID starting with 3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac not found: ID does not exist" containerID="3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac"
Sep 10 21:15:42.221685 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.221630    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac"} err="failed to get container status \"3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac\": rpc error: code = NotFound desc = could not find container \"3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac\": container with ID starting with 3b346dd2db8a433b13a2a7d1415f2d7d8bd0d77f8cbcde9c981a3b97f7e453ac not found: ID does not exist"
Sep 10 21:15:42.223278 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:42.223257    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-ac8722\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="6f535429-538e-4a75-a96c-fd543d26baab" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:42.459497 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.459474    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7734-0411ef"]
Sep 10 21:15:42.463897 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:42.463866    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69d9cb73_4f57_4fad_a9f2_4f1037ebe586.slice/crio-6bb665e45f38ab7f7a8268e64bb79750a7a670552f515376de37407cd4efe75e WatchSource:0}: Error finding container 6bb665e45f38ab7f7a8268e64bb79750a7a670552f515376de37407cd4efe75e: Status 404 returned error can't find the container with id 6bb665e45f38ab7f7a8268e64bb79750a7a670552f515376de37407cd4efe75e
Sep 10 21:15:42.617378 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:42.617346    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-ac8722\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="6f535429-538e-4a75-a96c-fd543d26baab" pod="opendatahub/test-curl-7734-ac8722"
Sep 10 21:15:42.618729 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:42.618708    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f535429-538e-4a75-a96c-fd543d26baab" path="/var/lib/kubelet/pods/6f535429-538e-4a75-a96c-fd543d26baab/volumes"
Sep 10 21:15:43.218514 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.218486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-0411ef" event={"ID":"69d9cb73-4f57-4fad-a9f2-4f1037ebe586","Type":"ContainerStarted","Data":"49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f"}
Sep 10 21:15:43.218888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.218530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7734-0411ef" event={"ID":"69d9cb73-4f57-4fad-a9f2-4f1037ebe586","Type":"ContainerStarted","Data":"6bb665e45f38ab7f7a8268e64bb79750a7a670552f515376de37407cd4efe75e"}
Sep 10 21:15:43.219638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.219612    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7734-4d39b5" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" containerName="test-curl-7734-4d39b5" containerID="cri-o://0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e" gracePeriod=2
Sep 10 21:15:43.232639 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.232596    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7734-0411ef" podStartSLOduration=2.2325819239999998 podStartE2EDuration="2.232581924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:43.231837024 +0000 UTC m=+1071.152457442" watchObservedRunningTime="2026-09-10 21:15:43.232581924 +0000 UTC m=+1071.153202310"
Sep 10 21:15:43.384681 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.384659    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:43.480981 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.480908    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2nnm\" (UniqueName: \"kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm\") pod \"2d4e1105-80b5-46ba-b3e9-3b8913265605\" (UID: \"2d4e1105-80b5-46ba-b3e9-3b8913265605\") "
Sep 10 21:15:43.482884 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.482860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm" (OuterVolumeSpecName: "kube-api-access-m2nnm") pod "2d4e1105-80b5-46ba-b3e9-3b8913265605" (UID: "2d4e1105-80b5-46ba-b3e9-3b8913265605"). InnerVolumeSpecName "kube-api-access-m2nnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:43.581396 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.581370    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nnm\" (UniqueName: \"kubernetes.io/projected/2d4e1105-80b5-46ba-b3e9-3b8913265605-kube-api-access-m2nnm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:15:43.794702 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.794617    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7734-0411ef"]
Sep 10 21:15:43.801652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:43.801625    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7734-0411ef"]
Sep 10 21:15:44.225554 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.225520    2562 generic.go:356] "Generic (PLEG): container finished" podID="2d4e1105-80b5-46ba-b3e9-3b8913265605" containerID="3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8" exitCode=137
Sep 10 21:15:44.225987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.225556    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:44.225987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.225618    2562 scope.go:117] "RemoveContainer" containerID="3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8"
Sep 10 21:15:44.225987 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:44.225947    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7734-0411ef" container="test-curl-7734-0411ef"
Sep 10 21:15:44.234183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.234166    2562 scope.go:117] "RemoveContainer" containerID="3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8"
Sep 10 21:15:44.234453 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:44.234437    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8\": container with ID starting with 3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8 not found: ID does not exist" containerID="3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8"
Sep 10 21:15:44.234520 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.234461    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8"} err="failed to get container status \"3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8\": rpc error: code = NotFound desc = could not find container \"3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8\": container with ID starting with 3bc27a5df5fc3d456bd3e850f1a0c750afd65c870f21f8f0f361823515df05d8 not found: ID does not exist"
Sep 10 21:15:44.235644 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:44.235625    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-c978c8\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" pod="opendatahub/test-curl-7734-c978c8"
Sep 10 21:15:44.622343 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:44.622236    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4e1105-80b5-46ba-b3e9-3b8913265605" path="/var/lib/kubelet/pods/2d4e1105-80b5-46ba-b3e9-3b8913265605/volumes"
Sep 10 21:15:45.232496 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:45.232444    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7734-0411ef" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" containerName="test-curl-7734-0411ef" containerID="cri-o://49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f" gracePeriod=2
Sep 10 21:15:45.395077 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:45.395053    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:45.494915 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:45.494845    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlw8l\" (UniqueName: \"kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l\") pod \"7580acc5-269f-423a-98ba-cc87f505c40d\" (UID: \"7580acc5-269f-423a-98ba-cc87f505c40d\") "
Sep 10 21:15:45.496752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:45.496724    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l" (OuterVolumeSpecName: "kube-api-access-tlw8l") pod "7580acc5-269f-423a-98ba-cc87f505c40d" (UID: "7580acc5-269f-423a-98ba-cc87f505c40d"). InnerVolumeSpecName "kube-api-access-tlw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:45.595600 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:45.595576    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlw8l\" (UniqueName: \"kubernetes.io/projected/7580acc5-269f-423a-98ba-cc87f505c40d-kube-api-access-tlw8l\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:15:46.238526 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.238490    2562 generic.go:356] "Generic (PLEG): container finished" podID="7580acc5-269f-423a-98ba-cc87f505c40d" containerID="0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e" exitCode=137
Sep 10 21:15:46.238965 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.238531    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:46.238965 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.238591    2562 scope.go:117] "RemoveContainer" containerID="0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e"
Sep 10 21:15:46.248119 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.248101    2562 scope.go:117] "RemoveContainer" containerID="0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e"
Sep 10 21:15:46.248399 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:46.248377    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e\": container with ID starting with 0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e not found: ID does not exist" containerID="0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e"
Sep 10 21:15:46.248484 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.248406    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e"} err="failed to get container status \"0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e\": rpc error: code = NotFound desc = could not find container \"0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e\": container with ID starting with 0dabd7e909505adcd39db263662ec1cee2eaae9a488db48f5f510fc0ccc9829e not found: ID does not exist"
Sep 10 21:15:46.250016 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:46.249993    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-4d39b5\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" pod="opendatahub/test-curl-7734-4d39b5"
Sep 10 21:15:46.618976 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:46.618897    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7580acc5-269f-423a-98ba-cc87f505c40d" path="/var/lib/kubelet/pods/7580acc5-269f-423a-98ba-cc87f505c40d/volumes"
Sep 10 21:15:47.411334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:47.411309    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:47.511748 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:47.511682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcg7g\" (UniqueName: \"kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g\") pod \"69d9cb73-4f57-4fad-a9f2-4f1037ebe586\" (UID: \"69d9cb73-4f57-4fad-a9f2-4f1037ebe586\") "
Sep 10 21:15:47.513631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:47.513602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g" (OuterVolumeSpecName: "kube-api-access-lcg7g") pod "69d9cb73-4f57-4fad-a9f2-4f1037ebe586" (UID: "69d9cb73-4f57-4fad-a9f2-4f1037ebe586"). InnerVolumeSpecName "kube-api-access-lcg7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:47.612657 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:47.612632    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcg7g\" (UniqueName: \"kubernetes.io/projected/69d9cb73-4f57-4fad-a9f2-4f1037ebe586-kube-api-access-lcg7g\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:15:48.251113 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.251076    2562 generic.go:356] "Generic (PLEG): container finished" podID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" containerID="49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f" exitCode=137
Sep 10 21:15:48.251113 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.251113    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:48.251336 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.251132    2562 scope.go:117] "RemoveContainer" containerID="49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f"
Sep 10 21:15:48.260260 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.260243    2562 scope.go:117] "RemoveContainer" containerID="49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f"
Sep 10 21:15:48.260508 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:48.260490    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f\": container with ID starting with 49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f not found: ID does not exist" containerID="49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f"
Sep 10 21:15:48.260551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.260518    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f"} err="failed to get container status \"49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f\": rpc error: code = NotFound desc = could not find container \"49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f\": container with ID starting with 49d54565ba0d59a3f977e5793e9cc29b1fcf785945541170a13d68bd0151cc1f not found: ID does not exist"
Sep 10 21:15:48.262202 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:15:48.262182    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7734-0411ef\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" pod="opendatahub/test-curl-7734-0411ef"
Sep 10 21:15:48.619872 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:48.619782    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" path="/var/lib/kubelet/pods/69d9cb73-4f57-4fad-a9f2-4f1037ebe586/volumes"
Sep 10 21:15:49.786559 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.786527    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:15:49.786924 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.786863    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" containerName="test-curl-7734-0411ef"
Sep 10 21:15:49.786924 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.786875    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" containerName="test-curl-7734-0411ef"
Sep 10 21:15:49.787003 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.786952    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d9cb73-4f57-4fad-a9f2-4f1037ebe586" containerName="test-curl-7734-0411ef"
Sep 10 21:15:49.791057 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.791039    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.795986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.795965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8ba2d470\""
Sep 10 21:15:49.796135 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.796048    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8ba2d470-dockercfg-png96\""
Sep 10 21:15:49.800042 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.800021    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:15:49.812024 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.811995    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:15:49.815126 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.815110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:49.827508 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.827490    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:15:49.831483 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.831454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.831588 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.831508    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.867542 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.867510    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:49.867712 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.867695    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:49.879904 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.879883    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:49.932141 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.932110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.932265 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.932149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.932265 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.932195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:49.932265 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.932220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrpt\" (UniqueName: \"kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:49.932920 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.932902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:49.939652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:49.939629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s\") pod \"payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:50.032899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.032875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrpt\" (UniqueName: \"kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:50.032998 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.032953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:50.033500 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.033484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:50.040546 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.040492    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrpt\" (UniqueName: \"kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt\") pod \"payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") " pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:50.101824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.101778    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:50.127389 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.127366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:50.272122 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.272096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:15:50.372531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.372507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:15:50.374058 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:50.374035    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1743fade_e2f9_4067_9af7_52a004c58678.slice/crio-fc1a231920fdf11ec0c767db0b1c28dbed08cfe939ceb1e46a1444f242d4567b WatchSource:0}: Error finding container fc1a231920fdf11ec0c767db0b1c28dbed08cfe939ceb1e46a1444f242d4567b: Status 404 returned error can't find the container with id fc1a231920fdf11ec0c767db0b1c28dbed08cfe939ceb1e46a1444f242d4567b
Sep 10 21:15:50.400342 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:50.400321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:15:50.404690 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:15:50.404669    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f331071_c007_476a_ad03_9d4103c1935f.slice/crio-8f86d408dbbf6c9f7bf33954bc94e11262336155b66ee3d13d8be91d4fdc6c6b WatchSource:0}: Error finding container 8f86d408dbbf6c9f7bf33954bc94e11262336155b66ee3d13d8be91d4fdc6c6b: Status 404 returned error can't find the container with id 8f86d408dbbf6c9f7bf33954bc94e11262336155b66ee3d13d8be91d4fdc6c6b
Sep 10 21:15:51.267595 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.267562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" event={"ID":"1743fade-e2f9-4067-9af7-52a004c58678","Type":"ContainerStarted","Data":"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"}
Sep 10 21:15:51.268094 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.267616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" event={"ID":"1743fade-e2f9-4067-9af7-52a004c58678","Type":"ContainerStarted","Data":"fc1a231920fdf11ec0c767db0b1c28dbed08cfe939ceb1e46a1444f242d4567b"}
Sep 10 21:15:51.268094 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.267674    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:15:51.269213 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.269193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" event={"ID":"6f331071-c007-476a-ad03-9d4103c1935f","Type":"ContainerStarted","Data":"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"}
Sep 10 21:15:51.269286 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.269227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" event={"ID":"6f331071-c007-476a-ad03-9d4103c1935f","Type":"ContainerStarted","Data":"8f86d408dbbf6c9f7bf33954bc94e11262336155b66ee3d13d8be91d4fdc6c6b"}
Sep 10 21:15:51.269286 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.269249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:15:51.284269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.284224    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" podStartSLOduration=2.284207749 podStartE2EDuration="2.284207749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:51.283671023 +0000 UTC m=+1079.204291422" watchObservedRunningTime="2026-09-10 21:15:51.284207749 +0000 UTC m=+1079.204828138"
Sep 10 21:15:51.300321 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:15:51.300278    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" podStartSLOduration=2.300265912 podStartE2EDuration="2.300265912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:51.298918169 +0000 UTC m=+1079.219538583" watchObservedRunningTime="2026-09-10 21:15:51.300265912 +0000 UTC m=+1079.220886297"
Sep 10 21:16:02.276501 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:02.276463    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:16:02.276976 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:02.276541    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:16:24.142631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.142599    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:16:24.146033 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.146015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.150011 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.149991    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"default-dockercfg-cz976\""
Sep 10 21:16:24.150011 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.150000    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"openshift-service-ca.crt\""
Sep 10 21:16:24.150168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.150084    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"auth-test-model-4eb314-kserve-self-signed-certs\""
Sep 10 21:16:24.150168 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.150089    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"kube-root-ca.crt\""
Sep 10 21:16:24.156062 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.156041    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:16:24.214997 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.214969    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzr6\" (UniqueName: \"kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.215136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.215012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.215136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.215047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.215136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.215074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.215136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.215107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.215304 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.215145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316107 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzr6\" (UniqueName: \"kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316444 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.316898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316873    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.317000 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.316882    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.318550 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.318529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.318725 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.318705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.324587 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.324563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzr6\" (UniqueName: \"kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6\") pod \"auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.456475 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.456450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:24.727975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:24.727938    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:16:24.729513 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:16:24.729481    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod890e1bd2_b185_4846_9946_f59fe8e6e2ea.slice/crio-d12372b690306ed0476e3c5100c2bdb8ba03eaf0108b2ac84094d0c44fc28842 WatchSource:0}: Error finding container d12372b690306ed0476e3c5100c2bdb8ba03eaf0108b2ac84094d0c44fc28842: Status 404 returned error can't find the container with id d12372b690306ed0476e3c5100c2bdb8ba03eaf0108b2ac84094d0c44fc28842
Sep 10 21:16:25.434846 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:25.434799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" event={"ID":"890e1bd2-b185-4846-9946-f59fe8e6e2ea","Type":"ContainerStarted","Data":"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"}
Sep 10 21:16:25.435183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:25.434854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" event={"ID":"890e1bd2-b185-4846-9946-f59fe8e6e2ea","Type":"ContainerStarted","Data":"d12372b690306ed0476e3c5100c2bdb8ba03eaf0108b2ac84094d0c44fc28842"}
Sep 10 21:16:25.451068 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:25.451025    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" podStartSLOduration=1.4510111239999999 podStartE2EDuration="1.451011124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:25.450235928 +0000 UTC m=+1113.370856326" watchObservedRunningTime="2026-09-10 21:16:25.451011124 +0000 UTC m=+1113.371631512"
Sep 10 21:16:34.457530 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:34.457490    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:34.457954 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:34.457547    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:34.470041 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:34.470012    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:16:34.488434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:16:34.488408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:17:17.698334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.698257    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"]
Sep 10 21:17:17.701723 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.701706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:17.704346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.704325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:17:17.704471 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.704330    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:17:17.715388 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.710385    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"]
Sep 10 21:17:17.790912 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.790883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8\" (UID: \"645f5e8a-fac1-4ece-be95-d59bfc133803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:17.892001 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.891973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8\" (UID: \"645f5e8a-fac1-4ece-be95-d59bfc133803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:17.892614 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:17.892591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8\" (UID: \"645f5e8a-fac1-4ece-be95-d59bfc133803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:18.014894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:18.014831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:18.286894 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:18.286863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"]
Sep 10 21:17:18.288878 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:17:18.288845    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645f5e8a_fac1_4ece_be95_d59bfc133803.slice/crio-e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187 WatchSource:0}: Error finding container e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187: Status 404 returned error can't find the container with id e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187
Sep 10 21:17:18.681392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:18.681358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8" event={"ID":"645f5e8a-fac1-4ece-be95-d59bfc133803","Type":"ContainerStarted","Data":"2c0f609585f3e4dd14df8f083938da09fb81adf594ed3971d6552815c5fc6578"}
Sep 10 21:17:18.681570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:18.681413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8" event={"ID":"645f5e8a-fac1-4ece-be95-d59bfc133803","Type":"ContainerStarted","Data":"e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187"}
Sep 10 21:17:18.696676 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:18.696631    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8" podStartSLOduration=1.696617058 podStartE2EDuration="1.696617058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:18.694439062 +0000 UTC m=+1166.615059448" watchObservedRunningTime="2026-09-10 21:17:18.696617058 +0000 UTC m=+1166.617237464"
Sep 10 21:17:19.687614 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:19.687579    2562 generic.go:356] "Generic (PLEG): container finished" podID="645f5e8a-fac1-4ece-be95-d59bfc133803" containerID="2c0f609585f3e4dd14df8f083938da09fb81adf594ed3971d6552815c5fc6578" exitCode=0
Sep 10 21:17:19.687999 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:19.687670    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8" event={"ID":"645f5e8a-fac1-4ece-be95-d59bfc133803","Type":"ContainerDied","Data":"2c0f609585f3e4dd14df8f083938da09fb81adf594ed3971d6552815c5fc6578"}
Sep 10 21:17:20.820960 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:20.820932    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:20.920143 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:20.920102    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca\") pod \"645f5e8a-fac1-4ece-be95-d59bfc133803\" (UID: \"645f5e8a-fac1-4ece-be95-d59bfc133803\") "
Sep 10 21:17:20.920470 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:20.920446    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "645f5e8a-fac1-4ece-be95-d59bfc133803" (UID: "645f5e8a-fac1-4ece-be95-d59bfc133803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:21.021591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:21.021505    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/645f5e8a-fac1-4ece-be95-d59bfc133803-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:17:21.698993 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:21.698962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"
Sep 10 21:17:21.698993 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:21.698980    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8" event={"ID":"645f5e8a-fac1-4ece-be95-d59bfc133803","Type":"ContainerDied","Data":"e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187"}
Sep 10 21:17:21.699207 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:21.699026    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e76bdc01a339029e3b31777534c867cd554b4ec354cd973a39d4001efdcd7187"
Sep 10 21:17:33.352407 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.352372    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:17:33.352972 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.352574    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" podUID="6f331071-c007-476a-ad03-9d4103c1935f" containerName="payload-processing" containerID="cri-o://e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c" gracePeriod=30
Sep 10 21:17:33.358238 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.358021    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:17:33.358383 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.358244    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" podUID="1743fade-e2f9-4067-9af7-52a004c58678" containerName="payload-pre-processing" containerID="cri-o://5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb" gracePeriod=30
Sep 10 21:17:33.521586 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.521509    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"]
Sep 10 21:17:33.523299 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.523272    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8ba2d470-l88m8"]
Sep 10 21:17:33.627328 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.627306    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:17:33.630573 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.630552    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:17:33.723700 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.723673    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phrpt\" (UniqueName: \"kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt\") pod \"6f331071-c007-476a-ad03-9d4103c1935f\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") "
Sep 10 21:17:33.723899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.723774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s\") pod \"1743fade-e2f9-4067-9af7-52a004c58678\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") "
Sep 10 21:17:33.723899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.723825    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume\") pod \"1743fade-e2f9-4067-9af7-52a004c58678\" (UID: \"1743fade-e2f9-4067-9af7-52a004c58678\") "
Sep 10 21:17:33.723899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.723882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume\") pod \"6f331071-c007-476a-ad03-9d4103c1935f\" (UID: \"6f331071-c007-476a-ad03-9d4103c1935f\") "
Sep 10 21:17:33.724250 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.724220    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1743fade-e2f9-4067-9af7-52a004c58678" (UID: "1743fade-e2f9-4067-9af7-52a004c58678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:33.724341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.724263    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f331071-c007-476a-ad03-9d4103c1935f" (UID: "6f331071-c007-476a-ad03-9d4103c1935f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:33.725911 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.725887    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt" (OuterVolumeSpecName: "kube-api-access-phrpt") pod "6f331071-c007-476a-ad03-9d4103c1935f" (UID: "6f331071-c007-476a-ad03-9d4103c1935f"). InnerVolumeSpecName "kube-api-access-phrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:33.726000 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.725953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s" (OuterVolumeSpecName: "kube-api-access-kph7s") pod "1743fade-e2f9-4067-9af7-52a004c58678" (UID: "1743fade-e2f9-4067-9af7-52a004c58678"). InnerVolumeSpecName "kube-api-access-kph7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:33.755132 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.755105    2562 generic.go:356] "Generic (PLEG): container finished" podID="1743fade-e2f9-4067-9af7-52a004c58678" containerID="5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb" exitCode=0
Sep 10 21:17:33.755240 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.755153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"
Sep 10 21:17:33.755240 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.755191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" event={"ID":"1743fade-e2f9-4067-9af7-52a004c58678","Type":"ContainerDied","Data":"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"}
Sep 10 21:17:33.755346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.755249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th" event={"ID":"1743fade-e2f9-4067-9af7-52a004c58678","Type":"ContainerDied","Data":"fc1a231920fdf11ec0c767db0b1c28dbed08cfe939ceb1e46a1444f242d4567b"}
Sep 10 21:17:33.755346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.755283    2562 scope.go:117] "RemoveContainer" containerID="5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"
Sep 10 21:17:33.756545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.756527    2562 generic.go:356] "Generic (PLEG): container finished" podID="6f331071-c007-476a-ad03-9d4103c1935f" containerID="e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c" exitCode=0
Sep 10 21:17:33.756638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.756554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" event={"ID":"6f331071-c007-476a-ad03-9d4103c1935f","Type":"ContainerDied","Data":"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"}
Sep 10 21:17:33.756638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.756578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"
Sep 10 21:17:33.756638 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.756599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d" event={"ID":"6f331071-c007-476a-ad03-9d4103c1935f","Type":"ContainerDied","Data":"8f86d408dbbf6c9f7bf33954bc94e11262336155b66ee3d13d8be91d4fdc6c6b"}
Sep 10 21:17:33.763707 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.763691    2562 scope.go:117] "RemoveContainer" containerID="5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"
Sep 10 21:17:33.764017 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:17:33.763995    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb\": container with ID starting with 5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb not found: ID does not exist" containerID="5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"
Sep 10 21:17:33.764100 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.764025    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb"} err="failed to get container status \"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb\": rpc error: code = NotFound desc = could not find container \"5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb\": container with ID starting with 5e2331ec6d77e31324ea298174adecc4989824e2cd75902f715cd69d25ce26bb not found: ID does not exist"
Sep 10 21:17:33.764100 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.764042    2562 scope.go:117] "RemoveContainer" containerID="e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"
Sep 10 21:17:33.772216 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.772191    2562 scope.go:117] "RemoveContainer" containerID="e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"
Sep 10 21:17:33.772496 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:17:33.772479    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c\": container with ID starting with e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c not found: ID does not exist" containerID="e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"
Sep 10 21:17:33.772549 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.772502    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c"} err="failed to get container status \"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c\": rpc error: code = NotFound desc = could not find container \"e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c\": container with ID starting with e789e7402e151c3cc69288fae6779179a2eb1b36b51869dcfd9603effef0858c not found: ID does not exist"
Sep 10 21:17:33.781757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.781737    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:17:33.785429 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.785408    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8ba2d470-67579b8c78-927th"]
Sep 10 21:17:33.795631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.795605    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:17:33.797880 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.797862    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8ba2d470-ddc66c59c-vrx7d"]
Sep 10 21:17:33.825258 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.825239    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrpt\" (UniqueName: \"kubernetes.io/projected/6f331071-c007-476a-ad03-9d4103c1935f-kube-api-access-phrpt\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:17:33.825258 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.825259    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kph7s\" (UniqueName: \"kubernetes.io/projected/1743fade-e2f9-4067-9af7-52a004c58678-kube-api-access-kph7s\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:17:33.825371 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.825269    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1743fade-e2f9-4067-9af7-52a004c58678-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:17:33.825371 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:33.825278    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f331071-c007-476a-ad03-9d4103c1935f-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:17:34.619366 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:34.619292    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1743fade-e2f9-4067-9af7-52a004c58678" path="/var/lib/kubelet/pods/1743fade-e2f9-4067-9af7-52a004c58678/volumes"
Sep 10 21:17:34.619701 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:34.619612    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645f5e8a-fac1-4ece-be95-d59bfc133803" path="/var/lib/kubelet/pods/645f5e8a-fac1-4ece-be95-d59bfc133803/volumes"
Sep 10 21:17:34.619933 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:17:34.619919    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f331071-c007-476a-ad03-9d4103c1935f" path="/var/lib/kubelet/pods/6f331071-c007-476a-ad03-9d4103c1935f/volumes"
Sep 10 21:18:34.930514 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930478    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-2ed8bf"]
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930793    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1743fade-e2f9-4067-9af7-52a004c58678" containerName="payload-pre-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930818    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1743fade-e2f9-4067-9af7-52a004c58678" containerName="payload-pre-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f331071-c007-476a-ad03-9d4103c1935f" containerName="payload-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930834    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f331071-c007-476a-ad03-9d4103c1935f" containerName="payload-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930841    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645f5e8a-fac1-4ece-be95-d59bfc133803" containerName="revoke-keys"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930846    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="645f5e8a-fac1-4ece-be95-d59bfc133803" containerName="revoke-keys"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930915    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1743fade-e2f9-4067-9af7-52a004c58678" containerName="payload-pre-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930922    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f331071-c007-476a-ad03-9d4103c1935f" containerName="payload-processing"
Sep 10 21:18:34.930990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.930930    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="645f5e8a-fac1-4ece-be95-d59bfc133803" containerName="revoke-keys"
Sep 10 21:18:34.933655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.933638    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:34.936650 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.936626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8md8h\""
Sep 10 21:18:34.939989 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:34.939965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-2ed8bf"]
Sep 10 21:18:35.029674 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:35.029638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9hwh\" (UniqueName: \"kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh\") pod \"test-curl-7737-2ed8bf\" (UID: \"531ee307-d124-44c0-97ff-ebaf01b7b7cd\") " pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:35.130901 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:35.130859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9hwh\" (UniqueName: \"kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh\") pod \"test-curl-7737-2ed8bf\" (UID: \"531ee307-d124-44c0-97ff-ebaf01b7b7cd\") " pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:35.139217 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:35.139185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9hwh\" (UniqueName: \"kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh\") pod \"test-curl-7737-2ed8bf\" (UID: \"531ee307-d124-44c0-97ff-ebaf01b7b7cd\") " pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:35.243486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:35.243452    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:35.509558 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:35.509527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-2ed8bf"]
Sep 10 21:18:35.511308 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:18:35.511266    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod531ee307_d124_44c0_97ff_ebaf01b7b7cd.slice/crio-5cb33cbaac4f5eaf2f8a9f3b5b767d254342de83e776d9e7039e0303143f0016 WatchSource:0}: Error finding container 5cb33cbaac4f5eaf2f8a9f3b5b767d254342de83e776d9e7039e0303143f0016: Status 404 returned error can't find the container with id 5cb33cbaac4f5eaf2f8a9f3b5b767d254342de83e776d9e7039e0303143f0016
Sep 10 21:18:36.049878 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:36.049841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-2ed8bf" event={"ID":"531ee307-d124-44c0-97ff-ebaf01b7b7cd","Type":"ContainerStarted","Data":"7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec"}
Sep 10 21:18:36.049878 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:36.049883    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-2ed8bf" event={"ID":"531ee307-d124-44c0-97ff-ebaf01b7b7cd","Type":"ContainerStarted","Data":"5cb33cbaac4f5eaf2f8a9f3b5b767d254342de83e776d9e7039e0303143f0016"}
Sep 10 21:18:36.065329 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:36.065285    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7737-2ed8bf" podStartSLOduration=2.06527116 podStartE2EDuration="2.06527116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:36.064652143 +0000 UTC m=+1243.985272530" watchObservedRunningTime="2026-09-10 21:18:36.06527116 +0000 UTC m=+1243.985891545"
Sep 10 21:18:36.624228 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:36.624192    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-2ed8bf"]
Sep 10 21:18:36.626037 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:36.626009    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-2ed8bf"]
Sep 10 21:18:37.055254 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:18:37.055224    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7737-2ed8bf" container="test-curl-7737-2ed8bf"
Sep 10 21:18:37.969819 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:37.969785    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:18:38.060194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:38.060133    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7737-2ed8bf" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" containerName="test-curl-7737-2ed8bf" containerID="cri-o://7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec" gracePeriod=2
Sep 10 21:18:38.064342 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:18:38.064313    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-2ed8bf\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:39.048495 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:39.048463    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:18:40.233172 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:40.233142    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:40.374374 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:40.374271    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9hwh\" (UniqueName: \"kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh\") pod \"531ee307-d124-44c0-97ff-ebaf01b7b7cd\" (UID: \"531ee307-d124-44c0-97ff-ebaf01b7b7cd\") "
Sep 10 21:18:40.376240 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:40.376220    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh" (OuterVolumeSpecName: "kube-api-access-q9hwh") pod "531ee307-d124-44c0-97ff-ebaf01b7b7cd" (UID: "531ee307-d124-44c0-97ff-ebaf01b7b7cd"). InnerVolumeSpecName "kube-api-access-q9hwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:40.475422 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:40.475396    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9hwh\" (UniqueName: \"kubernetes.io/projected/531ee307-d124-44c0-97ff-ebaf01b7b7cd-kube-api-access-q9hwh\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:40.618465 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:40.618434    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" path="/var/lib/kubelet/pods/531ee307-d124-44c0-97ff-ebaf01b7b7cd/volumes"
Sep 10 21:18:41.075709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:41.075676    2562 generic.go:356] "Generic (PLEG): container finished" podID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" containerID="7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec" exitCode=137
Sep 10 21:18:41.075909 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:41.075713    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-2ed8bf"
Sep 10 21:18:41.076593 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:41.075770    2562 scope.go:117] "RemoveContainer" containerID="7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec"
Sep 10 21:18:41.084679 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:41.084658    2562 scope.go:117] "RemoveContainer" containerID="7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec"
Sep 10 21:18:41.084964 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:18:41.084942    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec\": container with ID starting with 7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec not found: ID does not exist" containerID="7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec"
Sep 10 21:18:41.085039 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:41.084973    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec"} err="failed to get container status \"7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec\": rpc error: code = NotFound desc = could not find container \"7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec\": container with ID starting with 7fb9f69082b047959fafff97f312d4a13fac82eb6bb350296b7ac420f887f4ec not found: ID does not exist"
Sep 10 21:18:52.988955 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:52.988914    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" podUID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" containerName="main" containerID="cri-o://120eb803f50ac25c255872f128ef3e9ab2b8d1509585be55de90990bf7b643d5" gracePeriod=45
Sep 10 21:18:53.132462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.132278    2562 generic.go:356] "Generic (PLEG): container finished" podID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" containerID="120eb803f50ac25c255872f128ef3e9ab2b8d1509585be55de90990bf7b643d5" exitCode=0
Sep 10 21:18:53.132462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.132399    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" event={"ID":"12e3f345-4e63-4f6c-9281-14e40d9bd1c8","Type":"ContainerDied","Data":"120eb803f50ac25c255872f128ef3e9ab2b8d1509585be55de90990bf7b643d5"}
Sep 10 21:18:53.233135 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.233112    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:18:53.376551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376520    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-278h2\" (UniqueName: \"kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376545    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376845 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376845 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376845 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir\") pod \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\" (UID: \"12e3f345-4e63-4f6c-9281-14e40d9bd1c8\") "
Sep 10 21:18:53.376845 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376779    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home" (OuterVolumeSpecName: "home") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.377073 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.377073 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.376959    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.378878 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.378848    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm" (OuterVolumeSpecName: "dshm") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:53.378992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.378849    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:18:53.378992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.378880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2" (OuterVolumeSpecName: "kube-api-access-278h2") pod "12e3f345-4e63-4f6c-9281-14e40d9bd1c8" (UID: "12e3f345-4e63-4f6c-9281-14e40d9bd1c8"). InnerVolumeSpecName "kube-api-access-278h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:53.477899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477876    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.477899 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477897    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278h2\" (UniqueName: \"kubernetes.io/projected/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-kube-api-access-278h2\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.478047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477908    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.478047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477918    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.478047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477925    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:53.478047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:53.477948    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3f345-4e63-4f6c-9281-14e40d9bd1c8-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.067556 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.067487    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" podUID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" containerName="main" containerID="cri-o://a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d" gracePeriod=45
Sep 10 21:18:54.138513 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.138489    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"
Sep 10 21:18:54.138620 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.138513    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp" event={"ID":"12e3f345-4e63-4f6c-9281-14e40d9bd1c8","Type":"ContainerDied","Data":"873b00f00a971b1708d5ae49a6e6dacee90895d37f5c249594704e88dff564d1"}
Sep 10 21:18:54.138620 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.138554    2562 scope.go:117] "RemoveContainer" containerID="120eb803f50ac25c255872f128ef3e9ab2b8d1509585be55de90990bf7b643d5"
Sep 10 21:18:54.161242 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.161217    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:18:54.166685 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.166665    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/auth-test-model-8623b2-kserve-7887b7cc9-hlnwp"]
Sep 10 21:18:54.318918 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.318863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:18:54.385432 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385574 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hzr6\" (UniqueName: \"kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385574 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385495    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385574 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385522    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385574 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385574 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache\") pod \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\" (UID: \"890e1bd2-b185-4846-9946-f59fe8e6e2ea\") "
Sep 10 21:18:54.385869 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385789    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home" (OuterVolumeSpecName: "home") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:54.385923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385865    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:54.385923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.385879    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:54.387586 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.387551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm" (OuterVolumeSpecName: "dshm") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:18:54.387712 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.387617    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:18:54.387712 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.387641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6" (OuterVolumeSpecName: "kube-api-access-2hzr6") pod "890e1bd2-b185-4846-9946-f59fe8e6e2ea" (UID: "890e1bd2-b185-4846-9946-f59fe8e6e2ea"). InnerVolumeSpecName "kube-api-access-2hzr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:18:54.486716 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486690    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.486716 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486711    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.486874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486720    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/890e1bd2-b185-4846-9946-f59fe8e6e2ea-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.486874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486730    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.486874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486737    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/890e1bd2-b185-4846-9946-f59fe8e6e2ea-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.486874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.486744    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzr6\" (UniqueName: \"kubernetes.io/projected/890e1bd2-b185-4846-9946-f59fe8e6e2ea-kube-api-access-2hzr6\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:18:54.618821 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:54.618749    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" path="/var/lib/kubelet/pods/12e3f345-4e63-4f6c-9281-14e40d9bd1c8/volumes"
Sep 10 21:18:55.144518 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.144482    2562 generic.go:356] "Generic (PLEG): container finished" podID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" containerID="a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d" exitCode=0
Sep 10 21:18:55.144992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.144526    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"
Sep 10 21:18:55.144992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.144569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" event={"ID":"890e1bd2-b185-4846-9946-f59fe8e6e2ea","Type":"ContainerDied","Data":"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"}
Sep 10 21:18:55.144992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.144621    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q" event={"ID":"890e1bd2-b185-4846-9946-f59fe8e6e2ea","Type":"ContainerDied","Data":"d12372b690306ed0476e3c5100c2bdb8ba03eaf0108b2ac84094d0c44fc28842"}
Sep 10 21:18:55.144992 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.144645    2562 scope.go:117] "RemoveContainer" containerID="a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"
Sep 10 21:18:55.152696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.152660    2562 scope.go:117] "RemoveContainer" containerID="a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"
Sep 10 21:18:55.152970 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:18:55.152949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d\": container with ID starting with a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d not found: ID does not exist" containerID="a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"
Sep 10 21:18:55.153113 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.152976    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d"} err="failed to get container status \"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d\": rpc error: code = NotFound desc = could not find container \"a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d\": container with ID starting with a126bc703c3347c94651e618a3f7696a08b95768952b966c435b4c3e710e226d not found: ID does not exist"
Sep 10 21:18:55.160870 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.160848    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:18:55.164340 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:55.164315    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/auth-test-model-4eb314-kserve-79c8cf5d75-lqz6q"]
Sep 10 21:18:56.618452 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:18:56.618420    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" path="/var/lib/kubelet/pods/890e1bd2-b185-4846-9946-f59fe8e6e2ea/volumes"
Sep 10 21:19:09.567096 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567062    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567374    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" containerName="main"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567384    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" containerName="main"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567393    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" containerName="main"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567400    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" containerName="main"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567405    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" containerName="test-curl-7737-2ed8bf"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567411    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" containerName="test-curl-7737-2ed8bf"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567475    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e3f345-4e63-4f6c-9281-14e40d9bd1c8" containerName="main"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567485    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="531ee307-d124-44c0-97ff-ebaf01b7b7cd" containerName="test-curl-7737-2ed8bf"
Sep 10 21:19:09.567545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.567492    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="890e1bd2-b185-4846-9946-f59fe8e6e2ea" containerName="main"
Sep 10 21:19:09.572086 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.572067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.576097 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.576058    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"kube-root-ca.crt\""
Sep 10 21:19:09.576519 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.576369    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"sub-test-model-4eb314-kserve-self-signed-certs\""
Sep 10 21:19:09.576519 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.576393    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"default-dockercfg-cz976\""
Sep 10 21:19:09.576519 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.576393    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"openshift-service-ca.crt\""
Sep 10 21:19:09.579389 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.579365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:19:09.715309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.715469 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.715469 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.715469 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.715583 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.715583 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.715510    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mzm\" (UniqueName: \"kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816225 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mzm\" (UniqueName: \"kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816692 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816692 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.816692 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.816685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.818647 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.818595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.818906 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.818889    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.824534 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.824511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mzm\" (UniqueName: \"kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm\") pod \"sub-test-model-4eb314-kserve-67f79f65df-xlfrl\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:09.880996 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:09.880968    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:10.148021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:10.147991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:19:10.149387 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:19:10.149361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2b8a1f_6829_45d6_aef3_4f8cc8806d2d.slice/crio-81fdcdf2cbc6e7e66b04f711e6258c1358485dd96ad17005ccd5df3e373d5e58 WatchSource:0}: Error finding container 81fdcdf2cbc6e7e66b04f711e6258c1358485dd96ad17005ccd5df3e373d5e58: Status 404 returned error can't find the container with id 81fdcdf2cbc6e7e66b04f711e6258c1358485dd96ad17005ccd5df3e373d5e58
Sep 10 21:19:10.216035 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:10.216007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" event={"ID":"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d","Type":"ContainerStarted","Data":"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"}
Sep 10 21:19:10.216148 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:10.216046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" event={"ID":"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d","Type":"ContainerStarted","Data":"81fdcdf2cbc6e7e66b04f711e6258c1358485dd96ad17005ccd5df3e373d5e58"}
Sep 10 21:19:10.234849 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:10.234785    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" podStartSLOduration=1.234770192 podStartE2EDuration="1.234770192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:10.234474804 +0000 UTC m=+1278.155095207" watchObservedRunningTime="2026-09-10 21:19:10.234770192 +0000 UTC m=+1278.155390577"
Sep 10 21:19:19.882005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:19.881971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:19.882460 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:19.882025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:19.894523 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:19.894497    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:20.277612 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:20.277582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:19:22.256670 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.256633    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:19:22.259964 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.259943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.263081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.263057    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:19:22.263194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.263138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3a1a62-serving-cert\""
Sep 10 21:19:22.263194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.263155    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:19:22.264269 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.264252    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:19:22.264371 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.264314    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tww8l\""
Sep 10 21:19:22.271526 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.271504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:19:22.312340 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.312307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.312513 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.312353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.312513 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.312371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.312513 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.312492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7nc\" (UniqueName: \"kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.413575 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.413541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7nc\" (UniqueName: \"kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.413730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.413609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.413730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.413627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.413730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.413643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.414375 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.414351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.416091 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.416064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.416091 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.416079    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.424567 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.424547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7nc\" (UniqueName: \"kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc\") pod \"maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.571837 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.571757    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:22.842684 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.842658    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:19:22.844235 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:19:22.844206    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5154e5a_a52d_4c04_a7ab_bcfb57561508.slice/crio-cac841307814f500ee07b0db62ba4deb6c36fe3f42e01725c78c752cd04f4b37 WatchSource:0}: Error finding container cac841307814f500ee07b0db62ba4deb6c36fe3f42e01725c78c752cd04f4b37: Status 404 returned error can't find the container with id cac841307814f500ee07b0db62ba4deb6c36fe3f42e01725c78c752cd04f4b37
Sep 10 21:19:22.845504 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.845485    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:19:22.858474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.858451    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:19:22.862641 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.862623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:22.865136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.865118    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3a1a62\""
Sep 10 21:19:22.873070 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.873047    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:19:22.916649 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.916624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc7w\" (UniqueName: \"kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:22.916746 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:22.916700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.017596 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.017569    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc7w\" (UniqueName: \"kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.017734 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.017638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.018320 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.018301    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.028197 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.028179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc7w\" (UniqueName: \"kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w\") pod \"payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.174536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.174456    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:23.282635 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.282595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" event={"ID":"c5154e5a-a52d-4c04-a7ab-bcfb57561508","Type":"ContainerStarted","Data":"cac841307814f500ee07b0db62ba4deb6c36fe3f42e01725c78c752cd04f4b37"}
Sep 10 21:19:23.448563 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:23.448518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:19:23.449916 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:19:23.449883    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153cbf85_41a2_4c22_b49d_1b94990ac40a.slice/crio-07c514bf426381246cfb78e5385c90e322b0d8abd5187912e99ae5dcf1391ccd WatchSource:0}: Error finding container 07c514bf426381246cfb78e5385c90e322b0d8abd5187912e99ae5dcf1391ccd: Status 404 returned error can't find the container with id 07c514bf426381246cfb78e5385c90e322b0d8abd5187912e99ae5dcf1391ccd
Sep 10 21:19:24.291211 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:24.291176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" event={"ID":"153cbf85-41a2-4c22-b49d-1b94990ac40a","Type":"ContainerStarted","Data":"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"}
Sep 10 21:19:24.291635 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:24.291237    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" event={"ID":"153cbf85-41a2-4c22-b49d-1b94990ac40a","Type":"ContainerStarted","Data":"07c514bf426381246cfb78e5385c90e322b0d8abd5187912e99ae5dcf1391ccd"}
Sep 10 21:19:24.291635 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:24.291320    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:24.308590 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:24.308537    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" podStartSLOduration=2.30851639 podStartE2EDuration="2.30851639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:24.30613779 +0000 UTC m=+1292.226758195" watchObservedRunningTime="2026-09-10 21:19:24.30851639 +0000 UTC m=+1292.229136777"
Sep 10 21:19:25.298061 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:25.298031    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" event={"ID":"c5154e5a-a52d-4c04-a7ab-bcfb57561508","Type":"ContainerStarted","Data":"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"}
Sep 10 21:19:25.298441 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:25.298356    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:25.314373 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:25.314324    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" podStartSLOduration=0.956022558 podStartE2EDuration="3.314308511s" totalImagesPullingTime="2.358285953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:25.313299749 +0000 UTC m=+1293.233920173" watchObservedRunningTime="2026-09-10 21:19:25.314308511 +0000 UTC m=+1293.234928897"
Sep 10 21:19:31.308107 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:31.308070    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:19:35.300353 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:35.300320    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:19:51.284897 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.284864    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:19:51.288450 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.288435    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.291234 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.291212    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-044266-dockercfg-bmj6r\""
Sep 10 21:19:51.291381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.291219    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-044266\""
Sep 10 21:19:51.298876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.298852    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:19:51.308784 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.308763    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:19:51.312345 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.312325    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.322735 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.322711    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:19:51.374479 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.374454    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjhc\" (UniqueName: \"kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.374597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.374487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.475799 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.475767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.475962 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.475895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfvv4\" (UniqueName: \"kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.475962 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.475943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjhc\" (UniqueName: \"kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.476039 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.475964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.476689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.476672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.485145 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.485124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjhc\" (UniqueName: \"kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc\") pod \"payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.577111 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.577013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfvv4\" (UniqueName: \"kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.577111 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.577096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.577923 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.577897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.585153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.585131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfvv4\" (UniqueName: \"kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4\") pod \"payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") " pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.598905 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.598883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:51.622443 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.622421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:51.874357 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.874326    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:19:51.876084 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:19:51.876055    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3471caec_a78c_46d8_9cc9_6c8c79fae582.slice/crio-b2686c046220cdfec690f5269ab7f01236705bd5ea89acbe8e0a0e5f9c8b7430 WatchSource:0}: Error finding container b2686c046220cdfec690f5269ab7f01236705bd5ea89acbe8e0a0e5f9c8b7430: Status 404 returned error can't find the container with id b2686c046220cdfec690f5269ab7f01236705bd5ea89acbe8e0a0e5f9c8b7430
Sep 10 21:19:51.900170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:51.900146    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:19:51.907746 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:19:51.907720    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d78bd8_9b7b_455b_a7ae_89170e20a684.slice/crio-4622491e6803a7b3b7bb3d3390a0b54d79711523a59a3f6d237d709a3102f4e0 WatchSource:0}: Error finding container 4622491e6803a7b3b7bb3d3390a0b54d79711523a59a3f6d237d709a3102f4e0: Status 404 returned error can't find the container with id 4622491e6803a7b3b7bb3d3390a0b54d79711523a59a3f6d237d709a3102f4e0
Sep 10 21:19:52.433071 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.433042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" event={"ID":"d5d78bd8-9b7b-455b-a7ae-89170e20a684","Type":"ContainerStarted","Data":"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"}
Sep 10 21:19:52.433510 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.433088    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" event={"ID":"d5d78bd8-9b7b-455b-a7ae-89170e20a684","Type":"ContainerStarted","Data":"4622491e6803a7b3b7bb3d3390a0b54d79711523a59a3f6d237d709a3102f4e0"}
Sep 10 21:19:52.433510 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.433205    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:19:52.434716 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.434688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" event={"ID":"3471caec-a78c-46d8-9cc9-6c8c79fae582","Type":"ContainerStarted","Data":"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"}
Sep 10 21:19:52.434874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.434727    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" event={"ID":"3471caec-a78c-46d8-9cc9-6c8c79fae582","Type":"ContainerStarted","Data":"b2686c046220cdfec690f5269ab7f01236705bd5ea89acbe8e0a0e5f9c8b7430"}
Sep 10 21:19:52.434874 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.434763    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:19:52.449862 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.449802    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" podStartSLOduration=1.4497861699999999 podStartE2EDuration="1.44978617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:52.449178696 +0000 UTC m=+1320.369799102" watchObservedRunningTime="2026-09-10 21:19:52.44978617 +0000 UTC m=+1320.370406560"
Sep 10 21:19:52.465468 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:19:52.465426    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" podStartSLOduration=1.465412347 podStartE2EDuration="1.465412347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:52.463744139 +0000 UTC m=+1320.384364544" watchObservedRunningTime="2026-09-10 21:19:52.465412347 +0000 UTC m=+1320.386032733"
Sep 10 21:20:03.440941 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:03.440901    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:20:03.441404 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:03.440986    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:20:29.737858 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.737819    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-186c16"]
Sep 10 21:20:29.741625 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.741606    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:29.744194 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.744172    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8md8h\""
Sep 10 21:20:29.744781 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.744759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-186c16"]
Sep 10 21:20:29.802863 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.802834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9b4\" (UniqueName: \"kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4\") pod \"test-curl-7737-186c16\" (UID: \"73313e72-774f-4647-9597-92be98ffeff5\") " pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:29.903570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.903542    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9b4\" (UniqueName: \"kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4\") pod \"test-curl-7737-186c16\" (UID: \"73313e72-774f-4647-9597-92be98ffeff5\") " pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:29.911562 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:29.911533    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9b4\" (UniqueName: \"kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4\") pod \"test-curl-7737-186c16\" (UID: \"73313e72-774f-4647-9597-92be98ffeff5\") " pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:30.051429 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:30.051409    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:30.320286 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:30.320261    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-186c16"]
Sep 10 21:20:30.322249 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:20:30.322222    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73313e72_774f_4647_9597_92be98ffeff5.slice/crio-ad4c5f06d2bd7b47898502fca2b48d30aa1a460e1137cfc6345806829a6aa9b5 WatchSource:0}: Error finding container ad4c5f06d2bd7b47898502fca2b48d30aa1a460e1137cfc6345806829a6aa9b5: Status 404 returned error can't find the container with id ad4c5f06d2bd7b47898502fca2b48d30aa1a460e1137cfc6345806829a6aa9b5
Sep 10 21:20:30.618423 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:30.618334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-186c16" event={"ID":"73313e72-774f-4647-9597-92be98ffeff5","Type":"ContainerStarted","Data":"9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0"}
Sep 10 21:20:30.618423 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:30.618386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-186c16" event={"ID":"73313e72-774f-4647-9597-92be98ffeff5","Type":"ContainerStarted","Data":"ad4c5f06d2bd7b47898502fca2b48d30aa1a460e1137cfc6345806829a6aa9b5"}
Sep 10 21:20:30.629167 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:30.629122    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7737-186c16" podStartSLOduration=1.629108285 podStartE2EDuration="1.629108285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:30.627221822 +0000 UTC m=+1358.547842207" watchObservedRunningTime="2026-09-10 21:20:30.629108285 +0000 UTC m=+1358.549728670"
Sep 10 21:20:31.178441 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.178404    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-186c16"]
Sep 10 21:20:31.184579 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.184544    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-186c16"]
Sep 10 21:20:31.296834 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.296746    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-3abf24"]
Sep 10 21:20:31.297179 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.297160    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73313e72-774f-4647-9597-92be98ffeff5" containerName="test-curl-7737-186c16"
Sep 10 21:20:31.297260 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.297182    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="73313e72-774f-4647-9597-92be98ffeff5" containerName="test-curl-7737-186c16"
Sep 10 21:20:31.297315 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.297297    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="73313e72-774f-4647-9597-92be98ffeff5" containerName="test-curl-7737-186c16"
Sep 10 21:20:31.300285 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.300267    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:31.307198 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.307154    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-3abf24"]
Sep 10 21:20:31.416443 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.416418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdb2f\" (UniqueName: \"kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f\") pod \"test-curl-7737-3abf24\" (UID: \"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4\") " pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:31.517146 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.517121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdb2f\" (UniqueName: \"kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f\") pod \"test-curl-7737-3abf24\" (UID: \"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4\") " pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:31.525232 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.525206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdb2f\" (UniqueName: \"kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f\") pod \"test-curl-7737-3abf24\" (UID: \"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4\") " pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:31.611445 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.611368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:31.619283 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:31.619258    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7737-186c16" container="test-curl-7737-186c16"
Sep 10 21:20:31.879754 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:31.879682    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-3abf24"]
Sep 10 21:20:31.882910 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:20:31.882882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d38a3ba_e65e_4d52_96d0_4d6b1fe694d4.slice/crio-c5c4eb1a6c6637555e4a0659bc77413ef86958e82e89af2b5a7c5d6ce1d6f75d WatchSource:0}: Error finding container c5c4eb1a6c6637555e4a0659bc77413ef86958e82e89af2b5a7c5d6ce1d6f75d: Status 404 returned error can't find the container with id c5c4eb1a6c6637555e4a0659bc77413ef86958e82e89af2b5a7c5d6ce1d6f75d
Sep 10 21:20:32.624693 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.624650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-3abf24" event={"ID":"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4","Type":"ContainerStarted","Data":"ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1"}
Sep 10 21:20:32.624693 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.624697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-3abf24" event={"ID":"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4","Type":"ContainerStarted","Data":"c5c4eb1a6c6637555e4a0659bc77413ef86958e82e89af2b5a7c5d6ce1d6f75d"}
Sep 10 21:20:32.625170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.624712    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7737-186c16" podUID="73313e72-774f-4647-9597-92be98ffeff5" containerName="test-curl-7737-186c16" containerID="cri-o://9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0" gracePeriod=2
Sep 10 21:20:32.638821 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.638771    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7737-3abf24" podStartSLOduration=1.638757223 podStartE2EDuration="1.638757223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:32.637753453 +0000 UTC m=+1360.558373841" watchObservedRunningTime="2026-09-10 21:20:32.638757223 +0000 UTC m=+1360.559377608"
Sep 10 21:20:32.932035 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.931950    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:20:32.932228 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.932183    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" podUID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" containerName="maas-api" containerID="cri-o://b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea" gracePeriod=30
Sep 10 21:20:32.984608 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.984579    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:20:32.984904 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:32.984848    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" podUID="153cbf85-41a2-4c22-b49d-1b94990ac40a" containerName="payload-pre-processing" containerID="cri-o://1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8" gracePeriod=30
Sep 10 21:20:33.209640 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.209609    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-3abf24"]
Sep 10 21:20:33.216091 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.216062    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-3abf24"]
Sep 10 21:20:33.237350 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.237327    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:20:33.268973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.268948    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:20:33.334209 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334175    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca\") pod \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") "
Sep 10 21:20:33.334383 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdc7w\" (UniqueName: \"kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w\") pod \"153cbf85-41a2-4c22-b49d-1b94990ac40a\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") "
Sep 10 21:20:33.334383 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume\") pod \"153cbf85-41a2-4c22-b49d-1b94990ac40a\" (UID: \"153cbf85-41a2-4c22-b49d-1b94990ac40a\") "
Sep 10 21:20:33.334511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs\") pod \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") "
Sep 10 21:20:33.334511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334452    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq7nc\" (UniqueName: \"kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc\") pod \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") "
Sep 10 21:20:33.334511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334505    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls\") pod \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\" (UID: \"c5154e5a-a52d-4c04-a7ab-bcfb57561508\") "
Sep 10 21:20:33.334651 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334554    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5154e5a-a52d-4c04-a7ab-bcfb57561508" (UID: "c5154e5a-a52d-4c04-a7ab-bcfb57561508"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:33.334702 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334662    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "153cbf85-41a2-4c22-b49d-1b94990ac40a" (UID: "153cbf85-41a2-4c22-b49d-1b94990ac40a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:33.334830 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334792    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/153cbf85-41a2-4c22-b49d-1b94990ac40a-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.334903 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.334836    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5154e5a-a52d-4c04-a7ab-bcfb57561508-openshift-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.336708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.336684    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5154e5a-a52d-4c04-a7ab-bcfb57561508" (UID: "c5154e5a-a52d-4c04-a7ab-bcfb57561508"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:33.336795 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.336722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc" (OuterVolumeSpecName: "kube-api-access-mq7nc") pod "c5154e5a-a52d-4c04-a7ab-bcfb57561508" (UID: "c5154e5a-a52d-4c04-a7ab-bcfb57561508"). InnerVolumeSpecName "kube-api-access-mq7nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:33.336855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.336782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5154e5a-a52d-4c04-a7ab-bcfb57561508" (UID: "c5154e5a-a52d-4c04-a7ab-bcfb57561508"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:33.336855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.336791    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w" (OuterVolumeSpecName: "kube-api-access-fdc7w") pod "153cbf85-41a2-4c22-b49d-1b94990ac40a" (UID: "153cbf85-41a2-4c22-b49d-1b94990ac40a"). InnerVolumeSpecName "kube-api-access-fdc7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:33.436309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.436241    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-metrics-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.436309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.436267    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7nc\" (UniqueName: \"kubernetes.io/projected/c5154e5a-a52d-4c04-a7ab-bcfb57561508-kube-api-access-mq7nc\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.436309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.436278    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5154e5a-a52d-4c04-a7ab-bcfb57561508-maas-api-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.436309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.436287    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc7w\" (UniqueName: \"kubernetes.io/projected/153cbf85-41a2-4c22-b49d-1b94990ac40a-kube-api-access-fdc7w\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:33.630699 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.630666    2562 generic.go:356] "Generic (PLEG): container finished" podID="153cbf85-41a2-4c22-b49d-1b94990ac40a" containerID="1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8" exitCode=0
Sep 10 21:20:33.631162 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.630701    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"
Sep 10 21:20:33.631162 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.630758    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" event={"ID":"153cbf85-41a2-4c22-b49d-1b94990ac40a","Type":"ContainerDied","Data":"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"}
Sep 10 21:20:33.631162 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.630814    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll" event={"ID":"153cbf85-41a2-4c22-b49d-1b94990ac40a","Type":"ContainerDied","Data":"07c514bf426381246cfb78e5385c90e322b0d8abd5187912e99ae5dcf1391ccd"}
Sep 10 21:20:33.631162 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.630848    2562 scope.go:117] "RemoveContainer" containerID="1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"
Sep 10 21:20:33.632125 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.632085    2562 generic.go:356] "Generic (PLEG): container finished" podID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" containerID="b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea" exitCode=0
Sep 10 21:20:33.632186 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.632122    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"
Sep 10 21:20:33.632186 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.632155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" event={"ID":"c5154e5a-a52d-4c04-a7ab-bcfb57561508","Type":"ContainerDied","Data":"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"}
Sep 10 21:20:33.632284 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.632207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd" event={"ID":"c5154e5a-a52d-4c04-a7ab-bcfb57561508","Type":"ContainerDied","Data":"cac841307814f500ee07b0db62ba4deb6c36fe3f42e01725c78c752cd04f4b37"}
Sep 10 21:20:33.632391 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:33.632373    2562 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7737-3abf24" container="test-curl-7737-3abf24"
Sep 10 21:20:33.640978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.640963    2562 scope.go:117] "RemoveContainer" containerID="1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"
Sep 10 21:20:33.641250 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:33.641231    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8\": container with ID starting with 1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8 not found: ID does not exist" containerID="1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"
Sep 10 21:20:33.641323 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.641259    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8"} err="failed to get container status \"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8\": rpc error: code = NotFound desc = could not find container \"1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8\": container with ID starting with 1ececa791394a0bc7df1fc79f0ed3a25983265bd1f197e60be273579b25759b8 not found: ID does not exist"
Sep 10 21:20:33.641323 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.641277    2562 scope.go:117] "RemoveContainer" containerID="b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"
Sep 10 21:20:33.649375 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.649358    2562 scope.go:117] "RemoveContainer" containerID="b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"
Sep 10 21:20:33.649639 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:33.649622    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea\": container with ID starting with b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea not found: ID does not exist" containerID="b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"
Sep 10 21:20:33.649685 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.649646    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea"} err="failed to get container status \"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea\": rpc error: code = NotFound desc = could not find container \"b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea\": container with ID starting with b9ab435eba357b94c79ae1d7c0c58e223fde7565aba4b6b50097ce3424b5bfea not found: ID does not exist"
Sep 10 21:20:33.654178 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.654156    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:20:33.657340 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.657323    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3a1a62-7d596fcd66-5l8ll"]
Sep 10 21:20:33.667042 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.667019    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:20:33.671029 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:33.671010    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3a1a62-6fbbd76b69-lgkjd"]
Sep 10 21:20:34.619309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.619270    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153cbf85-41a2-4c22-b49d-1b94990ac40a" path="/var/lib/kubelet/pods/153cbf85-41a2-4c22-b49d-1b94990ac40a/volumes"
Sep 10 21:20:34.619658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.619644    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" path="/var/lib/kubelet/pods/c5154e5a-a52d-4c04-a7ab-bcfb57561508/volumes"
Sep 10 21:20:34.639891 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.639867    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7737-3abf24" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" containerName="test-curl-7737-3abf24" containerID="cri-o://ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1" gracePeriod=2
Sep 10 21:20:34.800615 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.800591    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:34.947990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.947959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz9b4\" (UniqueName: \"kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4\") pod \"73313e72-774f-4647-9597-92be98ffeff5\" (UID: \"73313e72-774f-4647-9597-92be98ffeff5\") "
Sep 10 21:20:34.950021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:34.949992    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4" (OuterVolumeSpecName: "kube-api-access-tz9b4") pod "73313e72-774f-4647-9597-92be98ffeff5" (UID: "73313e72-774f-4647-9597-92be98ffeff5"). InnerVolumeSpecName "kube-api-access-tz9b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:35.049559 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.049532    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz9b4\" (UniqueName: \"kubernetes.io/projected/73313e72-774f-4647-9597-92be98ffeff5-kube-api-access-tz9b4\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:35.645392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.645355    2562 generic.go:356] "Generic (PLEG): container finished" podID="73313e72-774f-4647-9597-92be98ffeff5" containerID="9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0" exitCode=137
Sep 10 21:20:35.645935 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.645398    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:35.645935 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.645450    2562 scope.go:117] "RemoveContainer" containerID="9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0"
Sep 10 21:20:35.654493 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.654472    2562 scope.go:117] "RemoveContainer" containerID="9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0"
Sep 10 21:20:35.654739 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:35.654718    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0\": container with ID starting with 9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0 not found: ID does not exist" containerID="9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0"
Sep 10 21:20:35.654785 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:35.654749    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0"} err="failed to get container status \"9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0\": rpc error: code = NotFound desc = could not find container \"9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0\": container with ID starting with 9704f9f8f13e5cc51b5341df8eb78f8230a710feb86e515f47539b5776bc6ae0 not found: ID does not exist"
Sep 10 21:20:35.656703 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:35.656682    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-186c16\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="73313e72-774f-4647-9597-92be98ffeff5" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:36.239689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:36.239659    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:20:36.256927 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:36.256885    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-186c16\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="73313e72-774f-4647-9597-92be98ffeff5" pod="opendatahub/test-curl-7737-186c16"
Sep 10 21:20:36.618077 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:36.618005    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73313e72-774f-4647-9597-92be98ffeff5" path="/var/lib/kubelet/pods/73313e72-774f-4647-9597-92be98ffeff5/volumes"
Sep 10 21:20:36.800039 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:36.800016    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:36.964031 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:36.963998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdb2f\" (UniqueName: \"kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f\") pod \"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4\" (UID: \"5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4\") "
Sep 10 21:20:36.966033 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:36.966002    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f" (OuterVolumeSpecName: "kube-api-access-rdb2f") pod "5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" (UID: "5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4"). InnerVolumeSpecName "kube-api-access-rdb2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:37.065144 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.065103    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdb2f\" (UniqueName: \"kubernetes.io/projected/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4-kube-api-access-rdb2f\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:37.657036 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.656999    2562 generic.go:356] "Generic (PLEG): container finished" podID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" containerID="ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1" exitCode=137
Sep 10 21:20:37.657036 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.657040    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:37.657287 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.657103    2562 scope.go:117] "RemoveContainer" containerID="ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1"
Sep 10 21:20:37.666021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.666005    2562 scope.go:117] "RemoveContainer" containerID="ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1"
Sep 10 21:20:37.666279 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:37.666259    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1\": container with ID starting with ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1 not found: ID does not exist" containerID="ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1"
Sep 10 21:20:37.666324 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:37.666288    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1"} err="failed to get container status \"ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1\": rpc error: code = NotFound desc = could not find container \"ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1\": container with ID starting with ff4ffef434e46f8d25e2e8d7b769ff2d5b73bc858446b4e83687c8fbc2c79ed1 not found: ID does not exist"
Sep 10 21:20:37.668469 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:37.668443    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-3abf24\" is forbidden: User \"system:node:ip-10-0-135-61.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-61.ec2.internal' and this object" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" pod="opendatahub/test-curl-7737-3abf24"
Sep 10 21:20:38.619053 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:38.619022    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" path="/var/lib/kubelet/pods/5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4/volumes"
Sep 10 21:20:39.558754 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.558719    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"]
Sep 10 21:20:39.559068 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559055    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" containerName="test-curl-7737-3abf24"
Sep 10 21:20:39.559116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559070    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" containerName="test-curl-7737-3abf24"
Sep 10 21:20:39.559116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559082    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" containerName="maas-api"
Sep 10 21:20:39.559116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559088    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" containerName="maas-api"
Sep 10 21:20:39.559116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559101    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153cbf85-41a2-4c22-b49d-1b94990ac40a" containerName="payload-pre-processing"
Sep 10 21:20:39.559116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559107    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="153cbf85-41a2-4c22-b49d-1b94990ac40a" containerName="payload-pre-processing"
Sep 10 21:20:39.559280 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559161    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d38a3ba-e65e-4d52-96d0-4d6b1fe694d4" containerName="test-curl-7737-3abf24"
Sep 10 21:20:39.559280 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559171    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="153cbf85-41a2-4c22-b49d-1b94990ac40a" containerName="payload-pre-processing"
Sep 10 21:20:39.559280 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.559178    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5154e5a-a52d-4c04-a7ab-bcfb57561508" containerName="maas-api"
Sep 10 21:20:39.563531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.563514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:39.565993 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.565969    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:20:39.566132 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.566111    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:20:39.569442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.569414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"]
Sep 10 21:20:39.686975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.686944    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-044266-22m9l\" (UID: \"55cbceb7-e2db-4c49-b290-118cc97b1db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:39.787473 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.787442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-044266-22m9l\" (UID: \"55cbceb7-e2db-4c49-b290-118cc97b1db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:39.788029 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.788011    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-044266-22m9l\" (UID: \"55cbceb7-e2db-4c49-b290-118cc97b1db1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:39.874727 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:39.874666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:40.145609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:40.145527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"]
Sep 10 21:20:40.148697 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:20:40.148651    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cbceb7_e2db_4c49_b290_118cc97b1db1.slice/crio-2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1 WatchSource:0}: Error finding container 2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1: Status 404 returned error can't find the container with id 2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1
Sep 10 21:20:40.673229 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:40.673196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l" event={"ID":"55cbceb7-e2db-4c49-b290-118cc97b1db1","Type":"ContainerStarted","Data":"ea574253fcccde2d5422053305bd996adc45da4fd2c76a76a684128ac1589299"}
Sep 10 21:20:40.673370 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:40.673248    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l" event={"ID":"55cbceb7-e2db-4c49-b290-118cc97b1db1","Type":"ContainerStarted","Data":"2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1"}
Sep 10 21:20:40.691479 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:40.691417    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l" podStartSLOduration=1.691403064 podStartE2EDuration="1.691403064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:40.688827542 +0000 UTC m=+1368.609447940" watchObservedRunningTime="2026-09-10 21:20:40.691403064 +0000 UTC m=+1368.612023450"
Sep 10 21:20:41.679878 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:41.679837    2562 generic.go:356] "Generic (PLEG): container finished" podID="55cbceb7-e2db-4c49-b290-118cc97b1db1" containerID="ea574253fcccde2d5422053305bd996adc45da4fd2c76a76a684128ac1589299" exitCode=0
Sep 10 21:20:41.680050 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:41.679917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l" event={"ID":"55cbceb7-e2db-4c49-b290-118cc97b1db1","Type":"ContainerDied","Data":"ea574253fcccde2d5422053305bd996adc45da4fd2c76a76a684128ac1589299"}
Sep 10 21:20:42.812598 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:42.812576    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:42.916439 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:42.916411    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca\") pod \"55cbceb7-e2db-4c49-b290-118cc97b1db1\" (UID: \"55cbceb7-e2db-4c49-b290-118cc97b1db1\") "
Sep 10 21:20:42.916728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:42.916707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55cbceb7-e2db-4c49-b290-118cc97b1db1" (UID: "55cbceb7-e2db-4c49-b290-118cc97b1db1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:43.017359 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:43.017328    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cbceb7-e2db-4c49-b290-118cc97b1db1-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:43.689759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:43.689721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l" event={"ID":"55cbceb7-e2db-4c49-b290-118cc97b1db1","Type":"ContainerDied","Data":"2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1"}
Sep 10 21:20:43.689953 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:43.689777    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2248d03aaa1d819d978c4e36e47e3b1b77d4d9bc02c3c2ab4b68a065cdc510f1"
Sep 10 21:20:43.689953 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:43.689730    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"
Sep 10 21:20:46.959103 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.959071    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:20:46.959477 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.959424    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cbceb7-e2db-4c49-b290-118cc97b1db1" containerName="revoke-keys"
Sep 10 21:20:46.959477 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.959434    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cbceb7-e2db-4c49-b290-118cc97b1db1" containerName="revoke-keys"
Sep 10 21:20:46.959547 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.959506    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cbceb7-e2db-4c49-b290-118cc97b1db1" containerName="revoke-keys"
Sep 10 21:20:46.962411 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.962393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:46.966342 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.966320    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"kube-root-ca.crt\""
Sep 10 21:20:46.966458 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.966321    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"openshift-service-ca.crt\""
Sep 10 21:20:46.966503 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.966321    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"default-dockercfg-t2sp2\""
Sep 10 21:20:46.966503 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.966329    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8623b2\"/\"rate-test-model-8623b2-kserve-self-signed-certs\""
Sep 10 21:20:46.972363 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:46.972342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:20:47.050147 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgm62\" (UniqueName: \"kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.050282 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.050282 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.050381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.050381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.050381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.050335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151519 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgm62\" (UniqueName: \"kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.151720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.151690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.152139 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.152109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.152266 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.152139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.152266 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.152177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.153889 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.153865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.154266 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.154246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.159712 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.159696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgm62\" (UniqueName: \"kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62\") pod \"rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") " pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.271586 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.271501    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:47.541346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.541321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:20:47.543105 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:20:47.543074    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c19741d_6040_4e0b_bce1_5fcd033f0945.slice/crio-1858d4d070ae2eba85ad47711c8954041a7dd961b19c05f9a949b33f01318ec0 WatchSource:0}: Error finding container 1858d4d070ae2eba85ad47711c8954041a7dd961b19c05f9a949b33f01318ec0: Status 404 returned error can't find the container with id 1858d4d070ae2eba85ad47711c8954041a7dd961b19c05f9a949b33f01318ec0
Sep 10 21:20:47.708697 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.708660    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" event={"ID":"5c19741d-6040-4e0b-bce1-5fcd033f0945","Type":"ContainerStarted","Data":"81eab7c94057f072003ea82f5ed5b3694818205c52d87d33b5fb4463aa32008e"}
Sep 10 21:20:47.708697 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.708702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" event={"ID":"5c19741d-6040-4e0b-bce1-5fcd033f0945","Type":"ContainerStarted","Data":"1858d4d070ae2eba85ad47711c8954041a7dd961b19c05f9a949b33f01318ec0"}
Sep 10 21:20:47.725032 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:47.724990    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" podStartSLOduration=1.724977466 podStartE2EDuration="1.724977466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:47.724438818 +0000 UTC m=+1375.645059225" watchObservedRunningTime="2026-09-10 21:20:47.724977466 +0000 UTC m=+1375.645597851"
Sep 10 21:20:51.258653 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.258604    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" podUID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" containerName="main" containerID="cri-o://1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a" gracePeriod=45
Sep 10 21:20:51.504901 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.504878    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:20:51.591138 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591138 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591348 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591348 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591348 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591185    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591348 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mzm\" (UniqueName: \"kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm\") pod \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\" (UID: \"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d\") "
Sep 10 21:20:51.591602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591449    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:51.591602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591464    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:51.591602 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.591499    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home" (OuterVolumeSpecName: "home") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:51.593272 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.593249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm" (OuterVolumeSpecName: "kube-api-access-t5mzm") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "kube-api-access-t5mzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:51.593674 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.593655    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm" (OuterVolumeSpecName: "dshm") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:51.593720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.593659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" (UID: "bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:51.692631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692596    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.692631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692624    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.692631 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692634    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.692864 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692643    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.692864 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692650    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.692864 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.692659    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mzm\" (UniqueName: \"kubernetes.io/projected/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d-kube-api-access-t5mzm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:51.731197 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.731171    2562 generic.go:356] "Generic (PLEG): container finished" podID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" containerID="1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a" exitCode=0
Sep 10 21:20:51.731326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.731225    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"
Sep 10 21:20:51.731326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.731247    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" event={"ID":"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d","Type":"ContainerDied","Data":"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"}
Sep 10 21:20:51.731326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.731277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl" event={"ID":"bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d","Type":"ContainerDied","Data":"81fdcdf2cbc6e7e66b04f711e6258c1358485dd96ad17005ccd5df3e373d5e58"}
Sep 10 21:20:51.731326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.731293    2562 scope.go:117] "RemoveContainer" containerID="1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"
Sep 10 21:20:51.740005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.739986    2562 scope.go:117] "RemoveContainer" containerID="1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"
Sep 10 21:20:51.740266 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:51.740248    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a\": container with ID starting with 1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a not found: ID does not exist" containerID="1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"
Sep 10 21:20:51.740305 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.740274    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a"} err="failed to get container status \"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a\": rpc error: code = NotFound desc = could not find container \"1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a\": container with ID starting with 1bc1dc9cfcd0091ff177a1af1c96c528d21896ccd793128413e20994e05c215a not found: ID does not exist"
Sep 10 21:20:51.751660 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.751638    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:20:51.756021 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:51.756000    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/sub-test-model-4eb314-kserve-67f79f65df-xlfrl"]
Sep 10 21:20:52.620009 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:52.619982    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" path="/var/lib/kubelet/pods/bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d/volumes"
Sep 10 21:20:52.634415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:52.634393    2562 scope.go:117] "RemoveContainer" containerID="de90c2ab3f44627ead654c626f5043816cef076ad1fc31a61e5ee96884338d88"
Sep 10 21:20:55.053346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.053310    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:20:55.053797 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.053592    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" podUID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" containerName="payload-processing" containerID="cri-o://5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38" gracePeriod=30
Sep 10 21:20:55.060932 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.060879    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:20:55.061411 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.061385    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" podUID="3471caec-a78c-46d8-9cc9-6c8c79fae582" containerName="payload-pre-processing" containerID="cri-o://8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b" gracePeriod=30
Sep 10 21:20:55.207567 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.207506    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"]
Sep 10 21:20:55.210381 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.210344    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-044266-22m9l"]
Sep 10 21:20:55.330347 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.330321    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:20:55.333029 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.333009    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:20:55.423524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.423496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gjhc\" (UniqueName: \"kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc\") pod \"3471caec-a78c-46d8-9cc9-6c8c79fae582\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") "
Sep 10 21:20:55.423658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.423546    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume\") pod \"3471caec-a78c-46d8-9cc9-6c8c79fae582\" (UID: \"3471caec-a78c-46d8-9cc9-6c8c79fae582\") "
Sep 10 21:20:55.423658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.423615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfvv4\" (UniqueName: \"kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4\") pod \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") "
Sep 10 21:20:55.423658 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.423651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume\") pod \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\" (UID: \"d5d78bd8-9b7b-455b-a7ae-89170e20a684\") "
Sep 10 21:20:55.424044 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.424018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3471caec-a78c-46d8-9cc9-6c8c79fae582" (UID: "3471caec-a78c-46d8-9cc9-6c8c79fae582"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:55.424147 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.424039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5d78bd8-9b7b-455b-a7ae-89170e20a684" (UID: "d5d78bd8-9b7b-455b-a7ae-89170e20a684"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:55.425634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.425613    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc" (OuterVolumeSpecName: "kube-api-access-8gjhc") pod "3471caec-a78c-46d8-9cc9-6c8c79fae582" (UID: "3471caec-a78c-46d8-9cc9-6c8c79fae582"). InnerVolumeSpecName "kube-api-access-8gjhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:55.425738 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.425674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4" (OuterVolumeSpecName: "kube-api-access-cfvv4") pod "d5d78bd8-9b7b-455b-a7ae-89170e20a684" (UID: "d5d78bd8-9b7b-455b-a7ae-89170e20a684"). InnerVolumeSpecName "kube-api-access-cfvv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:55.525166 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.525139    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5d78bd8-9b7b-455b-a7ae-89170e20a684-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:55.525166 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.525164    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjhc\" (UniqueName: \"kubernetes.io/projected/3471caec-a78c-46d8-9cc9-6c8c79fae582-kube-api-access-8gjhc\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:55.525310 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.525173    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3471caec-a78c-46d8-9cc9-6c8c79fae582-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:55.525310 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.525181    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfvv4\" (UniqueName: \"kubernetes.io/projected/d5d78bd8-9b7b-455b-a7ae-89170e20a684-kube-api-access-cfvv4\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:20:55.750318 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.750283    2562 generic.go:356] "Generic (PLEG): container finished" podID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" containerID="5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38" exitCode=0
Sep 10 21:20:55.750474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.750322    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"
Sep 10 21:20:55.750474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.750371    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" event={"ID":"d5d78bd8-9b7b-455b-a7ae-89170e20a684","Type":"ContainerDied","Data":"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"}
Sep 10 21:20:55.750474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.750434    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp" event={"ID":"d5d78bd8-9b7b-455b-a7ae-89170e20a684","Type":"ContainerDied","Data":"4622491e6803a7b3b7bb3d3390a0b54d79711523a59a3f6d237d709a3102f4e0"}
Sep 10 21:20:55.750474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.750467    2562 scope.go:117] "RemoveContainer" containerID="5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"
Sep 10 21:20:55.751689 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.751667    2562 generic.go:356] "Generic (PLEG): container finished" podID="3471caec-a78c-46d8-9cc9-6c8c79fae582" containerID="8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b" exitCode=0
Sep 10 21:20:55.751798 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.751718    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"
Sep 10 21:20:55.751798 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.751748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" event={"ID":"3471caec-a78c-46d8-9cc9-6c8c79fae582","Type":"ContainerDied","Data":"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"}
Sep 10 21:20:55.751798 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.751798    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks" event={"ID":"3471caec-a78c-46d8-9cc9-6c8c79fae582","Type":"ContainerDied","Data":"b2686c046220cdfec690f5269ab7f01236705bd5ea89acbe8e0a0e5f9c8b7430"}
Sep 10 21:20:55.761182 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.761163    2562 scope.go:117] "RemoveContainer" containerID="5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"
Sep 10 21:20:55.761444 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:55.761428    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38\": container with ID starting with 5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38 not found: ID does not exist" containerID="5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"
Sep 10 21:20:55.761492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.761453    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38"} err="failed to get container status \"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38\": rpc error: code = NotFound desc = could not find container \"5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38\": container with ID starting with 5f6ffe92cd20bd0478b109ec0a154f2f7f77e973032358d81cfbce524c375d38 not found: ID does not exist"
Sep 10 21:20:55.761492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.761468    2562 scope.go:117] "RemoveContainer" containerID="8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"
Sep 10 21:20:55.769544 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.769519    2562 scope.go:117] "RemoveContainer" containerID="8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"
Sep 10 21:20:55.769782 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:20:55.769764    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b\": container with ID starting with 8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b not found: ID does not exist" containerID="8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"
Sep 10 21:20:55.769852 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.769787    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b"} err="failed to get container status \"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b\": rpc error: code = NotFound desc = could not find container \"8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b\": container with ID starting with 8a96209f8774c39d64ce5f1107e412a8568a6cb6ceefa74c4c4903309629168b not found: ID does not exist"
Sep 10 21:20:55.790641 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.790616    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:20:55.795364 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.795344    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-044266-7c8d9895bf-c2dmp"]
Sep 10 21:20:55.819519 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.819496    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:20:55.827772 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:55.827751    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-044266-85d877cb98-rndks"]
Sep 10 21:20:56.618697 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:56.618664    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3471caec-a78c-46d8-9cc9-6c8c79fae582" path="/var/lib/kubelet/pods/3471caec-a78c-46d8-9cc9-6c8c79fae582/volumes"
Sep 10 21:20:56.619085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:56.619022    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cbceb7-e2db-4c49-b290-118cc97b1db1" path="/var/lib/kubelet/pods/55cbceb7-e2db-4c49-b290-118cc97b1db1/volumes"
Sep 10 21:20:56.619333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:56.619320    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" path="/var/lib/kubelet/pods/d5d78bd8-9b7b-455b-a7ae-89170e20a684/volumes"
Sep 10 21:20:57.272023 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:57.271993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:57.272264 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:57.272237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:57.285385 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:57.285363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:20:57.772916 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:20:57.772886    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:21:01.791892 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.791860    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"]
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792172    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" containerName="main"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792182    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" containerName="main"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3471caec-a78c-46d8-9cc9-6c8c79fae582" containerName="payload-pre-processing"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792207    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3471caec-a78c-46d8-9cc9-6c8c79fae582" containerName="payload-pre-processing"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792219    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" containerName="payload-processing"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" containerName="payload-processing"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792276    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3471caec-a78c-46d8-9cc9-6c8c79fae582" containerName="payload-pre-processing"
Sep 10 21:21:01.792281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792286    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5d78bd8-9b7b-455b-a7ae-89170e20a684" containerName="payload-processing"
Sep 10 21:21:01.792523 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.792293    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2b8a1f-6829-45d6-aef3-4f8cc8806d2d" containerName="main"
Sep 10 21:21:01.796781 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.796760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:01.799793 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.799769    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:21:01.799935 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.799773    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:21:01.805326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.805305    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"]
Sep 10 21:21:01.879742 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.879712    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g\" (UID: \"02e68401-d613-4175-9499-a234057fc63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:01.981130 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.981095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g\" (UID: \"02e68401-d613-4175-9499-a234057fc63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:01.981712 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:01.981690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g\" (UID: \"02e68401-d613-4175-9499-a234057fc63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:02.110568 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:02.110488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:02.376190 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:02.376164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"]
Sep 10 21:21:02.378341 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:21:02.378315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e68401_d613_4175_9499_a234057fc63a.slice/crio-13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27 WatchSource:0}: Error finding container 13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27: Status 404 returned error can't find the container with id 13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27
Sep 10 21:21:02.785220 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:02.785183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g" event={"ID":"02e68401-d613-4175-9499-a234057fc63a","Type":"ContainerStarted","Data":"795332b02a56920e9a3bf9cd9f02ec8a7bad7b40015bd814e602f922bb428cc5"}
Sep 10 21:21:02.785418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:02.785242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g" event={"ID":"02e68401-d613-4175-9499-a234057fc63a","Type":"ContainerStarted","Data":"13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27"}
Sep 10 21:21:02.800492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:02.800443    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g" podStartSLOduration=1.800429049 podStartE2EDuration="1.800429049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:02.799008088 +0000 UTC m=+1390.719628496" watchObservedRunningTime="2026-09-10 21:21:02.800429049 +0000 UTC m=+1390.721049434"
Sep 10 21:21:03.791722 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:03.791682    2562 generic.go:356] "Generic (PLEG): container finished" podID="02e68401-d613-4175-9499-a234057fc63a" containerID="795332b02a56920e9a3bf9cd9f02ec8a7bad7b40015bd814e602f922bb428cc5" exitCode=0
Sep 10 21:21:03.791911 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:03.791764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g" event={"ID":"02e68401-d613-4175-9499-a234057fc63a","Type":"ContainerDied","Data":"795332b02a56920e9a3bf9cd9f02ec8a7bad7b40015bd814e602f922bb428cc5"}
Sep 10 21:21:04.918718 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:04.918694    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:05.006244 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.006215    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca\") pod \"02e68401-d613-4175-9499-a234057fc63a\" (UID: \"02e68401-d613-4175-9499-a234057fc63a\") "
Sep 10 21:21:05.006551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.006529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02e68401-d613-4175-9499-a234057fc63a" (UID: "02e68401-d613-4175-9499-a234057fc63a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:05.106931 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.106865    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02e68401-d613-4175-9499-a234057fc63a-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.801815 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.801775    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"
Sep 10 21:21:05.801975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.801769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g" event={"ID":"02e68401-d613-4175-9499-a234057fc63a","Type":"ContainerDied","Data":"13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27"}
Sep 10 21:21:05.801975 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:05.801858    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13503888a900ad7d81d85282a19bb23a074bfe1baa8cb0be9e74271eeadfdc27"
Sep 10 21:21:12.273723 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.273677    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:21:12.274223 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.273981    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" podUID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" containerName="payload-processing" containerID="cri-o://b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6" gracePeriod=30
Sep 10 21:21:12.444591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.444561    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"]
Sep 10 21:21:12.446402 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.446376    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-91c25e-4t29g"]
Sep 10 21:21:12.523319 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.523297    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:21:12.569077 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.569009    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkr59\" (UniqueName: \"kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59\") pod \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") "
Sep 10 21:21:12.569077 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.569058    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume\") pod \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\" (UID: \"8f4c0db9-85d9-4f1e-b2a8-1349b0064363\") "
Sep 10 21:21:12.569435 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.569408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f4c0db9-85d9-4f1e-b2a8-1349b0064363" (UID: "8f4c0db9-85d9-4f1e-b2a8-1349b0064363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:12.571067 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.571048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59" (OuterVolumeSpecName: "kube-api-access-jkr59") pod "8f4c0db9-85d9-4f1e-b2a8-1349b0064363" (UID: "8f4c0db9-85d9-4f1e-b2a8-1349b0064363"). InnerVolumeSpecName "kube-api-access-jkr59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:12.619029 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.619004    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e68401-d613-4175-9499-a234057fc63a" path="/var/lib/kubelet/pods/02e68401-d613-4175-9499-a234057fc63a/volumes"
Sep 10 21:21:12.670462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.670437    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkr59\" (UniqueName: \"kubernetes.io/projected/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-kube-api-access-jkr59\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:12.670462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.670460    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f4c0db9-85d9-4f1e-b2a8-1349b0064363-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:12.833584 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.833499    2562 generic.go:356] "Generic (PLEG): container finished" podID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" containerID="b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6" exitCode=0
Sep 10 21:21:12.833584 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.833541    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"
Sep 10 21:21:12.833757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.833587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" event={"ID":"8f4c0db9-85d9-4f1e-b2a8-1349b0064363","Type":"ContainerDied","Data":"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"}
Sep 10 21:21:12.833757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.833640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw" event={"ID":"8f4c0db9-85d9-4f1e-b2a8-1349b0064363","Type":"ContainerDied","Data":"94db9676610a42f5df0b09ff2f69f79670026ee91cbe29efc0129417e6b29c86"}
Sep 10 21:21:12.833757 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.833668    2562 scope.go:117] "RemoveContainer" containerID="b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"
Sep 10 21:21:12.841753 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.841732    2562 scope.go:117] "RemoveContainer" containerID="b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"
Sep 10 21:21:12.842085 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:21:12.842064    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6\": container with ID starting with b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6 not found: ID does not exist" containerID="b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"
Sep 10 21:21:12.842161 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.842093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6"} err="failed to get container status \"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6\": rpc error: code = NotFound desc = could not find container \"b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6\": container with ID starting with b9e8d8a71ceab4965790c3133fb2c14a672550007177e492be315644384103a6 not found: ID does not exist"
Sep 10 21:21:12.852175 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.852156    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:21:12.856346 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:12.856327    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-91c25e-df4454f99-6dfpw"]
Sep 10 21:21:14.618583 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:14.618547    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" path="/var/lib/kubelet/pods/8f4c0db9-85d9-4f1e-b2a8-1349b0064363/volumes"
Sep 10 21:21:19.016013 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.015975    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"]
Sep 10 21:21:19.016392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016310    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e68401-d613-4175-9499-a234057fc63a" containerName="revoke-keys"
Sep 10 21:21:19.016392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016321    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e68401-d613-4175-9499-a234057fc63a" containerName="revoke-keys"
Sep 10 21:21:19.016392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016335    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" containerName="payload-processing"
Sep 10 21:21:19.016392 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016341    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" containerName="payload-processing"
Sep 10 21:21:19.016522 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016411    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4c0db9-85d9-4f1e-b2a8-1349b0064363" containerName="payload-processing"
Sep 10 21:21:19.016522 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.016422    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e68401-d613-4175-9499-a234057fc63a" containerName="revoke-keys"
Sep 10 21:21:19.020687 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.020670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:19.023341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.023318    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:21:19.023437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.023352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:21:19.027350 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.027326    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"]
Sep 10 21:21:19.123787 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.123756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2\" (UID: \"ec1e689a-b949-409d-8380-ada572fe325b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:19.224227 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.224203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2\" (UID: \"ec1e689a-b949-409d-8380-ada572fe325b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:19.224755 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.224735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2\" (UID: \"ec1e689a-b949-409d-8380-ada572fe325b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:19.333116 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.333034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:19.600312 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.600287    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"]
Sep 10 21:21:19.602409 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:21:19.602374    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1e689a_b949_409d_8380_ada572fe325b.slice/crio-ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc WatchSource:0}: Error finding container ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc: Status 404 returned error can't find the container with id ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc
Sep 10 21:21:19.867710 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.867679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2" event={"ID":"ec1e689a-b949-409d-8380-ada572fe325b","Type":"ContainerStarted","Data":"bc3664cb6fd6f0eceaabb507ce23c2448d688715562c7638ef06815a3a425f1a"}
Sep 10 21:21:19.867839 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.867726    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2" event={"ID":"ec1e689a-b949-409d-8380-ada572fe325b","Type":"ContainerStarted","Data":"ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc"}
Sep 10 21:21:19.883733 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:19.883693    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2" podStartSLOduration=1.883679092 podStartE2EDuration="1.883679092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:19.881902754 +0000 UTC m=+1407.802523161" watchObservedRunningTime="2026-09-10 21:21:19.883679092 +0000 UTC m=+1407.804299478"
Sep 10 21:21:20.873744 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:20.873715    2562 generic.go:356] "Generic (PLEG): container finished" podID="ec1e689a-b949-409d-8380-ada572fe325b" containerID="bc3664cb6fd6f0eceaabb507ce23c2448d688715562c7638ef06815a3a425f1a" exitCode=0
Sep 10 21:21:20.874131 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:20.873762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2" event={"ID":"ec1e689a-b949-409d-8380-ada572fe325b","Type":"ContainerDied","Data":"bc3664cb6fd6f0eceaabb507ce23c2448d688715562c7638ef06815a3a425f1a"}
Sep 10 21:21:22.007549 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.007524    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:22.148171 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.148091    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca\") pod \"ec1e689a-b949-409d-8380-ada572fe325b\" (UID: \"ec1e689a-b949-409d-8380-ada572fe325b\") "
Sep 10 21:21:22.148437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.148415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec1e689a-b949-409d-8380-ada572fe325b" (UID: "ec1e689a-b949-409d-8380-ada572fe325b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:22.249333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.249304    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1e689a-b949-409d-8380-ada572fe325b-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:22.884696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.884614    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"
Sep 10 21:21:22.884696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.884628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2" event={"ID":"ec1e689a-b949-409d-8380-ada572fe325b","Type":"ContainerDied","Data":"ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc"}
Sep 10 21:21:22.884696 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:22.884669    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce14ba1d6cc5e73d99c1e56e257362d7a990327019abee8735a8908421e93afc"
Sep 10 21:21:29.482695 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.482659    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:21:29.483187 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.482922    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" podUID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" containerName="payload-pre-processing" containerID="cri-o://2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45" gracePeriod=30
Sep 10 21:21:29.658698 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.658668    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"]
Sep 10 21:21:29.661101 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.661074    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-837e16-h2vx2"]
Sep 10 21:21:29.743721 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.743670    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:21:29.815333 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.815304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82qgt\" (UniqueName: \"kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt\") pod \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") "
Sep 10 21:21:29.815465 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.815401    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume\") pod \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\" (UID: \"3f2f3442-70d7-413d-b180-c4dbf0a97f0d\") "
Sep 10 21:21:29.815740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.815708    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f2f3442-70d7-413d-b180-c4dbf0a97f0d" (UID: "3f2f3442-70d7-413d-b180-c4dbf0a97f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:29.817305 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.817282    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt" (OuterVolumeSpecName: "kube-api-access-82qgt") pod "3f2f3442-70d7-413d-b180-c4dbf0a97f0d" (UID: "3f2f3442-70d7-413d-b180-c4dbf0a97f0d"). InnerVolumeSpecName "kube-api-access-82qgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:29.916086 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.916056    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.916086 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.916081    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qgt\" (UniqueName: \"kubernetes.io/projected/3f2f3442-70d7-413d-b180-c4dbf0a97f0d-kube-api-access-82qgt\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.919074 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.919044    2562 generic.go:356] "Generic (PLEG): container finished" podID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" containerID="2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45" exitCode=0
Sep 10 21:21:29.919185 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.919091    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"
Sep 10 21:21:29.919185 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.919128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" event={"ID":"3f2f3442-70d7-413d-b180-c4dbf0a97f0d","Type":"ContainerDied","Data":"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"}
Sep 10 21:21:29.919185 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.919180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw" event={"ID":"3f2f3442-70d7-413d-b180-c4dbf0a97f0d","Type":"ContainerDied","Data":"a241ea8e70ac655b3890630572d377c4fa2d3bfcbf3ba5738083d4b9eabe88e9"}
Sep 10 21:21:29.919296 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.919203    2562 scope.go:117] "RemoveContainer" containerID="2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"
Sep 10 21:21:29.928063 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.928045    2562 scope.go:117] "RemoveContainer" containerID="2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"
Sep 10 21:21:29.928320 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:21:29.928302    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45\": container with ID starting with 2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45 not found: ID does not exist" containerID="2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"
Sep 10 21:21:29.928367 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.928329    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45"} err="failed to get container status \"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45\": rpc error: code = NotFound desc = could not find container \"2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45\": container with ID starting with 2b50abc162cbf817ec4ba8a2fdfb6f783b2d1bf50df5e89511a5404167ff7a45 not found: ID does not exist"
Sep 10 21:21:29.940542 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.940519    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:21:29.944485 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:29.944467    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-837e16-9d88bd654-lzkdw"]
Sep 10 21:21:30.618748 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:30.618712    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" path="/var/lib/kubelet/pods/3f2f3442-70d7-413d-b180-c4dbf0a97f0d/volumes"
Sep 10 21:21:30.619247 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:30.619210    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1e689a-b949-409d-8380-ada572fe325b" path="/var/lib/kubelet/pods/ec1e689a-b949-409d-8380-ada572fe325b/volumes"
Sep 10 21:21:38.863186 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863146    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863647    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1e689a-b949-409d-8380-ada572fe325b" containerName="revoke-keys"
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863667    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1e689a-b949-409d-8380-ada572fe325b" containerName="revoke-keys"
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863681    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" containerName="payload-pre-processing"
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863687    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" containerName="payload-pre-processing"
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863738    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1e689a-b949-409d-8380-ada572fe325b" containerName="revoke-keys"
Sep 10 21:21:38.863752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.863749    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f2f3442-70d7-413d-b180-c4dbf0a97f0d" containerName="payload-pre-processing"
Sep 10 21:21:38.867872 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.867850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.870692 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.870673    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"kube-root-ca.crt\""
Sep 10 21:21:38.871930 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.871914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"rate-test-model-4eb314-kserve-self-signed-certs\""
Sep 10 21:21:38.871990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.871918    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"openshift-service-ca.crt\""
Sep 10 21:21:38.872205 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.872192    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-4eb314\"/\"default-dockercfg-cz976\""
Sep 10 21:21:38.875327 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.875302    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:21:38.988569 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988542    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.988745 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.988745 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988670    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dxt\" (UniqueName: \"kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.988745 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.988939 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988754    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:38.988939 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:38.988790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090042 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dxt\" (UniqueName: \"kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090502 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090559 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.090559 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.090547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.092500 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.092482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.092607 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.092588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.098094 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.098075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dxt\" (UniqueName: \"kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt\") pod \"rate-test-model-4eb314-kserve-55858db795-xs5pb\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") " pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.177725 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.177666    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:39.446625 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.446597    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:21:39.448362 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:21:39.448332    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c3ac1f1_d656_4cd3_8371_45029f017ff7.slice/crio-cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84 WatchSource:0}: Error finding container cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84: Status 404 returned error can't find the container with id cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84
Sep 10 21:21:39.967771 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.967736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" event={"ID":"0c3ac1f1-d656-4cd3-8371-45029f017ff7","Type":"ContainerStarted","Data":"ddd6a7667d6fb01a730dcc725a4c5ff801076f85e8784c20b15e5c2407669e0c"}
Sep 10 21:21:39.968177 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.967783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" event={"ID":"0c3ac1f1-d656-4cd3-8371-45029f017ff7","Type":"ContainerStarted","Data":"cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84"}
Sep 10 21:21:39.985435 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:39.985391    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" podStartSLOduration=1.985374821 podStartE2EDuration="1.985374821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:39.983220832 +0000 UTC m=+1427.903841237" watchObservedRunningTime="2026-09-10 21:21:39.985374821 +0000 UTC m=+1427.905995207"
Sep 10 21:21:49.178498 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:49.178418    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:49.178915 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:49.178640    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:49.190972 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:49.190949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:21:50.025051 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:21:50.025021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:22:14.960601 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:14.960568    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:22:15.947490 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:15.947454    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:22:21.260492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.260448    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:22:21.265938 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.265890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.268675 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.268654    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:22:21.268987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.268685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3dfe0c-serving-cert\""
Sep 10 21:22:21.268987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.268861    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:22:21.268987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.268701    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tww8l\""
Sep 10 21:22:21.270484 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.270462    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:22:21.272983 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.272955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:22:21.343119 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.343076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2cn\" (UniqueName: \"kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.343294 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.343129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.343294 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.343242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.343294 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.343276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.443871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.443789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2cn\" (UniqueName: \"kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.443871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.443879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.444155 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.443932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.444155 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.443950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.444564 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.444542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.446407 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.446385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.446509 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.446455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.452541 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.452517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2cn\" (UniqueName: \"kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn\") pod \"maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.577692 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.577591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:21.852617 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:21.852590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:22:21.854343 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:22:21.854310    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a616e0_918a_4e92_9fcb_ea8d43126240.slice/crio-698bd7225b6fa231c43cfca6e81c9ef2d2499ac89e524c55db9e990cc4b66aa3 WatchSource:0}: Error finding container 698bd7225b6fa231c43cfca6e81c9ef2d2499ac89e524c55db9e990cc4b66aa3: Status 404 returned error can't find the container with id 698bd7225b6fa231c43cfca6e81c9ef2d2499ac89e524c55db9e990cc4b66aa3
Sep 10 21:22:22.156370 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:22.156288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" event={"ID":"c1a616e0-918a-4e92-9fcb-ea8d43126240","Type":"ContainerStarted","Data":"698bd7225b6fa231c43cfca6e81c9ef2d2499ac89e524c55db9e990cc4b66aa3"}
Sep 10 21:22:23.162531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:23.162500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" event={"ID":"c1a616e0-918a-4e92-9fcb-ea8d43126240","Type":"ContainerStarted","Data":"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"}
Sep 10 21:22:23.162934 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:23.162616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:23.180334 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:23.180283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" podStartSLOduration=1.817581538 podStartE2EDuration="2.180270564s" totalImagesPullingTime="362.689026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:23.17844942 +0000 UTC m=+1471.099069821" watchObservedRunningTime="2026-09-10 21:22:23.180270564 +0000 UTC m=+1471.100890986"
Sep 10 21:22:29.171999 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:29.171970    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:22:29.979521 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:29.979480    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" podUID="5c19741d-6040-4e0b-bce1-5fcd033f0945" containerName="main" containerID="cri-o://81eab7c94057f072003ea82f5ed5b3694818205c52d87d33b5fb4463aa32008e" gracePeriod=45
Sep 10 21:22:30.194785 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.194756    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c19741d-6040-4e0b-bce1-5fcd033f0945" containerID="81eab7c94057f072003ea82f5ed5b3694818205c52d87d33b5fb4463aa32008e" exitCode=0
Sep 10 21:22:30.195136 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.194847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" event={"ID":"5c19741d-6040-4e0b-bce1-5fcd033f0945","Type":"ContainerDied","Data":"81eab7c94057f072003ea82f5ed5b3694818205c52d87d33b5fb4463aa32008e"}
Sep 10 21:22:30.229946 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.229892    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:22:30.324270 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324313    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324341    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgm62\" (UniqueName: \"kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324362    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324387    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm\") pod \"5c19741d-6040-4e0b-bce1-5fcd033f0945\" (UID: \"5c19741d-6040-4e0b-bce1-5fcd033f0945\") "
Sep 10 21:22:30.324682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home" (OuterVolumeSpecName: "home") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:30.324682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324579    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:30.324682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324656    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:30.324782 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324691    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.324782 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.324710    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.326515 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.326487    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm" (OuterVolumeSpecName: "dshm") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:30.326515 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.326501    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62" (OuterVolumeSpecName: "kube-api-access-fgm62") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "kube-api-access-fgm62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:30.326672 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.326526    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c19741d-6040-4e0b-bce1-5fcd033f0945" (UID: "5c19741d-6040-4e0b-bce1-5fcd033f0945"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:22:30.425329 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.425300    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgm62\" (UniqueName: \"kubernetes.io/projected/5c19741d-6040-4e0b-bce1-5fcd033f0945-kube-api-access-fgm62\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.425329 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.425325    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c19741d-6040-4e0b-bce1-5fcd033f0945-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.425486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.425337    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.425486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.425345    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c19741d-6040-4e0b-bce1-5fcd033f0945-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:30.965898 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:30.965842    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" podUID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" containerName="main" containerID="cri-o://ddd6a7667d6fb01a730dcc725a4c5ff801076f85e8784c20b15e5c2407669e0c" gracePeriod=45
Sep 10 21:22:31.201103 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.201074    2562 generic.go:356] "Generic (PLEG): container finished" podID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" containerID="ddd6a7667d6fb01a730dcc725a4c5ff801076f85e8784c20b15e5c2407669e0c" exitCode=0
Sep 10 21:22:31.201437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.201104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" event={"ID":"0c3ac1f1-d656-4cd3-8371-45029f017ff7","Type":"ContainerDied","Data":"ddd6a7667d6fb01a730dcc725a4c5ff801076f85e8784c20b15e5c2407669e0c"}
Sep 10 21:22:31.201437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.201145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb" event={"ID":"0c3ac1f1-d656-4cd3-8371-45029f017ff7","Type":"ContainerDied","Data":"cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84"}
Sep 10 21:22:31.201437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.201160    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb80f5780c13edb0abe6ab434ca86f62bb655d47b6486e71540fb7880f954d84"
Sep 10 21:22:31.202498 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.202479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9" event={"ID":"5c19741d-6040-4e0b-bce1-5fcd033f0945","Type":"ContainerDied","Data":"1858d4d070ae2eba85ad47711c8954041a7dd961b19c05f9a949b33f01318ec0"}
Sep 10 21:22:31.202609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.202513    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"
Sep 10 21:22:31.202609 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.202520    2562 scope.go:117] "RemoveContainer" containerID="81eab7c94057f072003ea82f5ed5b3694818205c52d87d33b5fb4463aa32008e"
Sep 10 21:22:31.211739 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.211726    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:22:31.221324 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.221280    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:22:31.224703 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.224683    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8623b2/rate-test-model-8623b2-kserve-6c6cfd9c5f-2r2w9"]
Sep 10 21:22:31.332278 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332252    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332364    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65dxt\" (UniqueName: \"kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332391    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332461 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs\") pod \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\" (UID: \"0c3ac1f1-d656-4cd3-8371-45029f017ff7\") "
Sep 10 21:22:31.332740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home" (OuterVolumeSpecName: "home") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:31.332740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332666    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:31.332958 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.332893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:31.333190 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.333111    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:31.334685 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.334662    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:22:31.335015 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.334982    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm" (OuterVolumeSpecName: "dshm") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:22:31.335015 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.334988    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt" (OuterVolumeSpecName: "kube-api-access-65dxt") pod "0c3ac1f1-d656-4cd3-8371-45029f017ff7" (UID: "0c3ac1f1-d656-4cd3-8371-45029f017ff7"). InnerVolumeSpecName "kube-api-access-65dxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:31.433738 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.433709    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:31.433738 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.433734    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:31.433902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.433744    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dxt\" (UniqueName: \"kubernetes.io/projected/0c3ac1f1-d656-4cd3-8371-45029f017ff7-kube-api-access-65dxt\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:31.433902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.433754    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:31.433902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:31.433763    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c3ac1f1-d656-4cd3-8371-45029f017ff7-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:22:32.207979 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:32.207949    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"
Sep 10 21:22:32.228214 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:32.228183    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:22:32.230380 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:32.230359    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-4eb314/rate-test-model-4eb314-kserve-55858db795-xs5pb"]
Sep 10 21:22:32.623019 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:32.622941    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" path="/var/lib/kubelet/pods/0c3ac1f1-d656-4cd3-8371-45029f017ff7/volumes"
Sep 10 21:22:32.623321 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:32.623307    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c19741d-6040-4e0b-bce1-5fcd033f0945" path="/var/lib/kubelet/pods/5c19741d-6040-4e0b-bce1-5fcd033f0945/volumes"
Sep 10 21:22:50.455584 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.455550    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:22:50.455977 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.455956    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c19741d-6040-4e0b-bce1-5fcd033f0945" containerName="main"
Sep 10 21:22:50.455977 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.455974    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c19741d-6040-4e0b-bce1-5fcd033f0945" containerName="main"
Sep 10 21:22:50.456074 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.456010    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" containerName="main"
Sep 10 21:22:50.456074 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.456021    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" containerName="main"
Sep 10 21:22:50.456170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.456117    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3ac1f1-d656-4cd3-8371-45029f017ff7" containerName="main"
Sep 10 21:22:50.456170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.456134    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c19741d-6040-4e0b-bce1-5fcd033f0945" containerName="main"
Sep 10 21:22:50.460488 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.460468    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.463274 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.463257    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-636944-serving-cert\""
Sep 10 21:22:50.468586 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.468559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:22:50.492678 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.492652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5x4\" (UniqueName: \"kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.492796 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.492705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.492796 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.492764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.492932 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.492835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.593914 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.593889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5x4\" (UniqueName: \"kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.594023 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.593943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.594023 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.593968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.594023 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.593992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.594690 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.594670    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.596259 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.596237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.596349 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.596318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.603528 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.603505    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5x4\" (UniqueName: \"kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4\") pod \"maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:50.772946 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:50.772873    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:51.050202 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:51.050172    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:22:51.052344 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:22:51.052309    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32cbf91_5004_4610_88c9_a1e555c5c8d8.slice/crio-741eced9585026eef3dcd6d1d2cc5647ec9a2425adde214c954b35fb4a0e0f6c WatchSource:0}: Error finding container 741eced9585026eef3dcd6d1d2cc5647ec9a2425adde214c954b35fb4a0e0f6c: Status 404 returned error can't find the container with id 741eced9585026eef3dcd6d1d2cc5647ec9a2425adde214c954b35fb4a0e0f6c
Sep 10 21:22:51.298875 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:51.298837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" event={"ID":"e32cbf91-5004-4610-88c9-a1e555c5c8d8","Type":"ContainerStarted","Data":"741eced9585026eef3dcd6d1d2cc5647ec9a2425adde214c954b35fb4a0e0f6c"}
Sep 10 21:22:52.305183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:52.305146    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" event={"ID":"e32cbf91-5004-4610-88c9-a1e555c5c8d8","Type":"ContainerStarted","Data":"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"}
Sep 10 21:22:52.305555 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:52.305277    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:22:52.322440 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:52.322390    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" podStartSLOduration=1.8538438510000002 podStartE2EDuration="2.322369954s" totalImagesPullingTime="468.526103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:52.322295324 +0000 UTC m=+1500.242915721" watchObservedRunningTime="2026-09-10 21:22:52.322369954 +0000 UTC m=+1500.242990343"
Sep 10 21:22:58.316301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:22:58.316267    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:23:19.199119 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:19.199037    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:23:40.762960 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.762930    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:23:40.765986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.765971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.768849 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.768826    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3dfe0c\"/\"openshift-service-ca.crt\""
Sep 10 21:23:40.770424 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.770392    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3dfe0c\"/\"default-dockercfg-82pgd\""
Sep 10 21:23:40.770537 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.770449    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3dfe0c\"/\"ipp-route-3dfe0c-kserve-self-signed-certs\""
Sep 10 21:23:40.770537 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.770454    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3dfe0c\"/\"kube-root-ca.crt\""
Sep 10 21:23:40.776566 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.776542    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:23:40.891645 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.891645 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.891871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.891871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.891871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.891871 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.891855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdqs\" (UniqueName: \"kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992470 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992530    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdqs\" (UniqueName: \"kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.992884 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992635    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.993002 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.992978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.993080 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.993013    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.993080 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.993048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.994795 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.994779    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:40.995107 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:40.995091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:41.001046 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.001026    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdqs\" (UniqueName: \"kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs\") pod \"ipp-route-3dfe0c-kserve-b45cd6447-rnhvl\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") " pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:41.077081 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.077021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:41.345173 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.345149    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:23:41.347076 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:23:41.347048    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd674bdf8_00e7_4981_bf33_a2a931b4884e.slice/crio-1557cb0d7aae711205bd76d2a8f236cfa61e2307260b5673516b34edfb6ba528 WatchSource:0}: Error finding container 1557cb0d7aae711205bd76d2a8f236cfa61e2307260b5673516b34edfb6ba528: Status 404 returned error can't find the container with id 1557cb0d7aae711205bd76d2a8f236cfa61e2307260b5673516b34edfb6ba528
Sep 10 21:23:41.537318 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.537283    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" event={"ID":"d674bdf8-00e7-4981-bf33-a2a931b4884e","Type":"ContainerStarted","Data":"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"}
Sep 10 21:23:41.537495 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.537326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" event={"ID":"d674bdf8-00e7-4981-bf33-a2a931b4884e","Type":"ContainerStarted","Data":"1557cb0d7aae711205bd76d2a8f236cfa61e2307260b5673516b34edfb6ba528"}
Sep 10 21:23:41.554709 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:41.554658    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" podStartSLOduration=1.554646567 podStartE2EDuration="1.554646567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:41.553137547 +0000 UTC m=+1549.473757943" watchObservedRunningTime="2026-09-10 21:23:41.554646567 +0000 UTC m=+1549.475266953"
Sep 10 21:23:51.077296 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:51.077259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:51.077296 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:51.077303    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:51.089875 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:51.089843    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:51.597073 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:51.597040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:23:52.700910 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:23:52.700878    2562 scope.go:117] "RemoveContainer" containerID="2c0f609585f3e4dd14df8f083938da09fb81adf594ed3971d6552815c5fc6578"
Sep 10 21:24:05.141967 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:05.141927    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:24:09.970990 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:09.970952    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:09.974418 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:09.974394    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:09.977364 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:09.977347    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-361296-serving-cert\""
Sep 10 21:24:09.986551 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:09.986527    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:10.022219 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.022188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxt6j\" (UniqueName: \"kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.022414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.022258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.022414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.022356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.022414 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.022390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.122876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.122833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxt6j\" (UniqueName: \"kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.123062 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.122902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.123062 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.122934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.123167 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.123109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.123634 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.123608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.125341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.125320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.125469 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.125447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.130820 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.130784    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxt6j\" (UniqueName: \"kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j\") pod \"maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.285758 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.285667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:10.555045 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.555018    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:10.557651 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:24:10.557610    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdaa269_5442_451f_a604_6d3c31d61d87.slice/crio-6581aac8547348de3da42787d4eece92f5b39fccec7743c29ac9389a4c960a52 WatchSource:0}: Error finding container 6581aac8547348de3da42787d4eece92f5b39fccec7743c29ac9389a4c960a52: Status 404 returned error can't find the container with id 6581aac8547348de3da42787d4eece92f5b39fccec7743c29ac9389a4c960a52
Sep 10 21:24:10.679798 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:10.679765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" event={"ID":"ebdaa269-5442-451f-a604-6d3c31d61d87","Type":"ContainerStarted","Data":"6581aac8547348de3da42787d4eece92f5b39fccec7743c29ac9389a4c960a52"}
Sep 10 21:24:11.686870 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:11.686830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" event={"ID":"ebdaa269-5442-451f-a604-6d3c31d61d87","Type":"ContainerStarted","Data":"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"}
Sep 10 21:24:11.687246 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:11.686918    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:11.703641 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:11.703591    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" podStartSLOduration=2.239432505 podStartE2EDuration="2.703574506s" totalImagesPullingTime="464.142001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:11.701538575 +0000 UTC m=+1579.622158976" watchObservedRunningTime="2026-09-10 21:24:11.703574506 +0000 UTC m=+1579.624194891"
Sep 10 21:24:17.696411 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:17.696379    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:40.446818 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.446766    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:40.447368 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.446990    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" podUID="ebdaa269-5442-451f-a604-6d3c31d61d87" containerName="maas-api" containerID="cri-o://61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a" gracePeriod=30
Sep 10 21:24:40.688229 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.688204    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:40.779785 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.779714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs\") pod \"ebdaa269-5442-451f-a604-6d3c31d61d87\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") "
Sep 10 21:24:40.779785 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.779778    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxt6j\" (UniqueName: \"kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j\") pod \"ebdaa269-5442-451f-a604-6d3c31d61d87\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") "
Sep 10 21:24:40.780024 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.779800    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca\") pod \"ebdaa269-5442-451f-a604-6d3c31d61d87\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") "
Sep 10 21:24:40.780024 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.779837    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls\") pod \"ebdaa269-5442-451f-a604-6d3c31d61d87\" (UID: \"ebdaa269-5442-451f-a604-6d3c31d61d87\") "
Sep 10 21:24:40.780230 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.780190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebdaa269-5442-451f-a604-6d3c31d61d87" (UID: "ebdaa269-5442-451f-a604-6d3c31d61d87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:40.781855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.781829    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebdaa269-5442-451f-a604-6d3c31d61d87" (UID: "ebdaa269-5442-451f-a604-6d3c31d61d87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:40.781967 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.781851    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebdaa269-5442-451f-a604-6d3c31d61d87" (UID: "ebdaa269-5442-451f-a604-6d3c31d61d87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:40.781967 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.781866    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j" (OuterVolumeSpecName: "kube-api-access-rxt6j") pod "ebdaa269-5442-451f-a604-6d3c31d61d87" (UID: "ebdaa269-5442-451f-a604-6d3c31d61d87"). InnerVolumeSpecName "kube-api-access-rxt6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:40.827181 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.827146    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebdaa269-5442-451f-a604-6d3c31d61d87" containerID="61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a" exitCode=0
Sep 10 21:24:40.827301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.827194    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"
Sep 10 21:24:40.827301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.827221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" event={"ID":"ebdaa269-5442-451f-a604-6d3c31d61d87","Type":"ContainerDied","Data":"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"}
Sep 10 21:24:40.827301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.827274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p" event={"ID":"ebdaa269-5442-451f-a604-6d3c31d61d87","Type":"ContainerDied","Data":"6581aac8547348de3da42787d4eece92f5b39fccec7743c29ac9389a4c960a52"}
Sep 10 21:24:40.827301 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.827295    2562 scope.go:117] "RemoveContainer" containerID="61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"
Sep 10 21:24:40.836621 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.836601    2562 scope.go:117] "RemoveContainer" containerID="61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"
Sep 10 21:24:40.836906 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:24:40.836885    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a\": container with ID starting with 61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a not found: ID does not exist" containerID="61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"
Sep 10 21:24:40.837002 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.836912    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a"} err="failed to get container status \"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a\": rpc error: code = NotFound desc = could not find container \"61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a\": container with ID starting with 61d384dc2c28020a6d99f89b1cb5a7cba160fd19a07baad1e8248dfa902c526a not found: ID does not exist"
Sep 10 21:24:40.849439 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.849415    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:40.853309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.853289    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-361296-86bfb9bf5b-dzq2p"]
Sep 10 21:24:40.880653 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.880627    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxt6j\" (UniqueName: \"kubernetes.io/projected/ebdaa269-5442-451f-a604-6d3c31d61d87-kube-api-access-rxt6j\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:24:40.880653 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.880651    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebdaa269-5442-451f-a604-6d3c31d61d87-openshift-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:24:40.880779 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.880669    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-maas-api-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:24:40.880779 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:40.880683    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebdaa269-5442-451f-a604-6d3c31d61d87-metrics-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:24:42.618385 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:24:42.618349    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebdaa269-5442-451f-a604-6d3c31d61d87" path="/var/lib/kubelet/pods/ebdaa269-5442-451f-a604-6d3c31d61d87/volumes"
Sep 10 21:25:05.864775 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:05.864740    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:25:05.865480 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:05.865034    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" podUID="c1a616e0-918a-4e92-9fcb-ea8d43126240" containerName="maas-api" containerID="cri-o://6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4" gracePeriod=30
Sep 10 21:25:06.106163 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.106139    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:25:06.182787 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.182706    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs2cn\" (UniqueName: \"kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn\") pod \"c1a616e0-918a-4e92-9fcb-ea8d43126240\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") "
Sep 10 21:25:06.182787 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.182747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls\") pod \"c1a616e0-918a-4e92-9fcb-ea8d43126240\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") "
Sep 10 21:25:06.183002 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.182790    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs\") pod \"c1a616e0-918a-4e92-9fcb-ea8d43126240\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") "
Sep 10 21:25:06.183002 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.182859    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca\") pod \"c1a616e0-918a-4e92-9fcb-ea8d43126240\" (UID: \"c1a616e0-918a-4e92-9fcb-ea8d43126240\") "
Sep 10 21:25:06.183261 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.183235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1a616e0-918a-4e92-9fcb-ea8d43126240" (UID: "c1a616e0-918a-4e92-9fcb-ea8d43126240"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:06.184851 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.184825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1a616e0-918a-4e92-9fcb-ea8d43126240" (UID: "c1a616e0-918a-4e92-9fcb-ea8d43126240"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:25:06.184851 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.184835    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn" (OuterVolumeSpecName: "kube-api-access-vs2cn") pod "c1a616e0-918a-4e92-9fcb-ea8d43126240" (UID: "c1a616e0-918a-4e92-9fcb-ea8d43126240"). InnerVolumeSpecName "kube-api-access-vs2cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:06.184983 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.184949    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1a616e0-918a-4e92-9fcb-ea8d43126240" (UID: "c1a616e0-918a-4e92-9fcb-ea8d43126240"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:25:06.284034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.283996    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-metrics-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:06.284034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.284028    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a616e0-918a-4e92-9fcb-ea8d43126240-openshift-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:06.284241 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.284045    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2cn\" (UniqueName: \"kubernetes.io/projected/c1a616e0-918a-4e92-9fcb-ea8d43126240-kube-api-access-vs2cn\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:06.284241 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.284057    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a616e0-918a-4e92-9fcb-ea8d43126240-maas-api-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:06.956250 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.956217    2562 generic.go:356] "Generic (PLEG): container finished" podID="c1a616e0-918a-4e92-9fcb-ea8d43126240" containerID="6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4" exitCode=0
Sep 10 21:25:06.956633 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.956267    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"
Sep 10 21:25:06.956633 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.956284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" event={"ID":"c1a616e0-918a-4e92-9fcb-ea8d43126240","Type":"ContainerDied","Data":"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"}
Sep 10 21:25:06.956633 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.956337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd" event={"ID":"c1a616e0-918a-4e92-9fcb-ea8d43126240","Type":"ContainerDied","Data":"698bd7225b6fa231c43cfca6e81c9ef2d2499ac89e524c55db9e990cc4b66aa3"}
Sep 10 21:25:06.956633 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.956367    2562 scope.go:117] "RemoveContainer" containerID="6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"
Sep 10 21:25:06.964902 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.964877    2562 scope.go:117] "RemoveContainer" containerID="6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"
Sep 10 21:25:06.965163 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:25:06.965143    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4\": container with ID starting with 6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4 not found: ID does not exist" containerID="6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"
Sep 10 21:25:06.965230 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.965170    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4"} err="failed to get container status \"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4\": rpc error: code = NotFound desc = could not find container \"6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4\": container with ID starting with 6ddb533dea13f35a7cf94e4e03c946dae519774828187493db5662df86172eb4 not found: ID does not exist"
Sep 10 21:25:06.974206 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.974182    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:25:06.977305 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:06.977283    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3dfe0c-7f64dd4785-lvmrd"]
Sep 10 21:25:08.618933 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:08.618903    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a616e0-918a-4e92-9fcb-ea8d43126240" path="/var/lib/kubelet/pods/c1a616e0-918a-4e92-9fcb-ea8d43126240/volumes"
Sep 10 21:25:11.415374 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:11.415339    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:25:26.434429 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.434386    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" podUID="d674bdf8-00e7-4981-bf33-a2a931b4884e" containerName="main" containerID="cri-o://5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a" gracePeriod=45
Sep 10 21:25:26.681272 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.681248    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:25:26.745643 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.745824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.745824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfdqs\" (UniqueName: \"kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.745824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745790    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.745978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745842    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.745978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.745868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache\") pod \"d674bdf8-00e7-4981-bf33-a2a931b4884e\" (UID: \"d674bdf8-00e7-4981-bf33-a2a931b4884e\") "
Sep 10 21:25:26.746240 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.746188    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:26.746350 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.746233    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:26.746350 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.746220    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home" (OuterVolumeSpecName: "home") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:26.748351 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.748313    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs" (OuterVolumeSpecName: "kube-api-access-bfdqs") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "kube-api-access-bfdqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:26.748473 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.748375    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm" (OuterVolumeSpecName: "dshm") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:26.748473 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.748381    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d674bdf8-00e7-4981-bf33-a2a931b4884e" (UID: "d674bdf8-00e7-4981-bf33-a2a931b4884e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:25:26.846686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846658    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d674bdf8-00e7-4981-bf33-a2a931b4884e-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:26.846686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846681    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:26.846686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846691    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdqs\" (UniqueName: \"kubernetes.io/projected/d674bdf8-00e7-4981-bf33-a2a931b4884e-kube-api-access-bfdqs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:26.846886 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846700    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:26.846886 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846710    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:26.846886 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:26.846717    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d674bdf8-00e7-4981-bf33-a2a931b4884e-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:25:27.051637 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.051557    2562 generic.go:356] "Generic (PLEG): container finished" podID="d674bdf8-00e7-4981-bf33-a2a931b4884e" containerID="5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a" exitCode=0
Sep 10 21:25:27.051637 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.051597    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"
Sep 10 21:25:27.051637 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.051613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" event={"ID":"d674bdf8-00e7-4981-bf33-a2a931b4884e","Type":"ContainerDied","Data":"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"}
Sep 10 21:25:27.051867 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.051649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl" event={"ID":"d674bdf8-00e7-4981-bf33-a2a931b4884e","Type":"ContainerDied","Data":"1557cb0d7aae711205bd76d2a8f236cfa61e2307260b5673516b34edfb6ba528"}
Sep 10 21:25:27.051867 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.051667    2562 scope.go:117] "RemoveContainer" containerID="5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"
Sep 10 21:25:27.062543 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.062525    2562 scope.go:117] "RemoveContainer" containerID="5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"
Sep 10 21:25:27.062889 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:25:27.062871    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a\": container with ID starting with 5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a not found: ID does not exist" containerID="5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"
Sep 10 21:25:27.062941 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.062897    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a"} err="failed to get container status \"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a\": rpc error: code = NotFound desc = could not find container \"5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a\": container with ID starting with 5e6a35898256f83e6382a1be7d465b359d1d1f74d6bbe21d3c2a81771c5b605a not found: ID does not exist"
Sep 10 21:25:27.075153 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.075128    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:25:27.079341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:27.079321    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3dfe0c/ipp-route-3dfe0c-kserve-b45cd6447-rnhvl"]
Sep 10 21:25:28.619769 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:25:28.619726    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d674bdf8-00e7-4981-bf33-a2a931b4884e" path="/var/lib/kubelet/pods/d674bdf8-00e7-4981-bf33-a2a931b4884e/volumes"
Sep 10 21:26:05.244255 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.244221    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:26:05.244796 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.244438    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" podUID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" containerName="maas-api" containerID="cri-o://c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce" gracePeriod=30
Sep 10 21:26:05.484606 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.484580    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:26:05.662986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.662911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls\") pod \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") "
Sep 10 21:26:05.662986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.662956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca\") pod \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") "
Sep 10 21:26:05.662986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.662979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj5x4\" (UniqueName: \"kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4\") pod \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") "
Sep 10 21:26:05.663264 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.663001    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs\") pod \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\" (UID: \"e32cbf91-5004-4610-88c9-a1e555c5c8d8\") "
Sep 10 21:26:05.663409 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.663387    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e32cbf91-5004-4610-88c9-a1e555c5c8d8" (UID: "e32cbf91-5004-4610-88c9-a1e555c5c8d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:05.665143 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.665109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e32cbf91-5004-4610-88c9-a1e555c5c8d8" (UID: "e32cbf91-5004-4610-88c9-a1e555c5c8d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:05.665244 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.665202    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4" (OuterVolumeSpecName: "kube-api-access-sj5x4") pod "e32cbf91-5004-4610-88c9-a1e555c5c8d8" (UID: "e32cbf91-5004-4610-88c9-a1e555c5c8d8"). InnerVolumeSpecName "kube-api-access-sj5x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:05.665244 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.665211    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e32cbf91-5004-4610-88c9-a1e555c5c8d8" (UID: "e32cbf91-5004-4610-88c9-a1e555c5c8d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:05.764570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.764530    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-maas-api-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:05.764570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.764562    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32cbf91-5004-4610-88c9-a1e555c5c8d8-openshift-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:05.764570 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.764574    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5x4\" (UniqueName: \"kubernetes.io/projected/e32cbf91-5004-4610-88c9-a1e555c5c8d8-kube-api-access-sj5x4\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:05.764876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:05.764583    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32cbf91-5004-4610-88c9-a1e555c5c8d8-metrics-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:06.231691 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.231657    2562 generic.go:356] "Generic (PLEG): container finished" podID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" containerID="c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce" exitCode=0
Sep 10 21:26:06.231888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.231694    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"
Sep 10 21:26:06.231888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.231722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" event={"ID":"e32cbf91-5004-4610-88c9-a1e555c5c8d8","Type":"ContainerDied","Data":"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"}
Sep 10 21:26:06.231888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.231771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f" event={"ID":"e32cbf91-5004-4610-88c9-a1e555c5c8d8","Type":"ContainerDied","Data":"741eced9585026eef3dcd6d1d2cc5647ec9a2425adde214c954b35fb4a0e0f6c"}
Sep 10 21:26:06.231888 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.231793    2562 scope.go:117] "RemoveContainer" containerID="c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"
Sep 10 21:26:06.241597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.241569    2562 scope.go:117] "RemoveContainer" containerID="c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"
Sep 10 21:26:06.241886 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:26:06.241867    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce\": container with ID starting with c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce not found: ID does not exist" containerID="c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"
Sep 10 21:26:06.241946 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.241895    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce"} err="failed to get container status \"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce\": rpc error: code = NotFound desc = could not find container \"c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce\": container with ID starting with c715069af9178cda2fe3feea3f95de7d3193559199f91273581761da22cc75ce not found: ID does not exist"
Sep 10 21:26:06.250654 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.250630    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:26:06.253513 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.253494    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-636944-85b97b58f9-b4r5f"]
Sep 10 21:26:06.623859 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:06.623755    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" path="/var/lib/kubelet/pods/e32cbf91-5004-4610-88c9-a1e555c5c8d8/volumes"
Sep 10 21:26:22.592005 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.591921    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"]
Sep 10 21:26:22.593536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.593494    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8623b2-29817915-8n8fs"]
Sep 10 21:26:22.619726 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.619692    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2296c82b-a27d-413a-8fed-2d18c209adfe" path="/var/lib/kubelet/pods/2296c82b-a27d-413a-8fed-2d18c209adfe/volumes"
Sep 10 21:26:22.620031 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.620014    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:26:22.620257 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.620231    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" podUID="b651fb2f-5918-4139-81b6-af25d3c0c151" containerName="payload-processing" containerID="cri-o://e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd" gracePeriod=30
Sep 10 21:26:22.621996 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.621969    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:26:22.622363 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.622317    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" podUID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" containerName="payload-pre-processing" containerID="cri-o://d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2" gracePeriod=30
Sep 10 21:26:22.885103 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.885078    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:26:22.887780 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:22.887761    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:26:23.001897 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.001866    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4xrz\" (UniqueName: \"kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz\") pod \"b651fb2f-5918-4139-81b6-af25d3c0c151\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") "
Sep 10 21:26:23.001897 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.001899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnm7m\" (UniqueName: \"kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m\") pod \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") "
Sep 10 21:26:23.002130 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.001926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume\") pod \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\" (UID: \"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0\") "
Sep 10 21:26:23.002130 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.001950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume\") pod \"b651fb2f-5918-4139-81b6-af25d3c0c151\" (UID: \"b651fb2f-5918-4139-81b6-af25d3c0c151\") "
Sep 10 21:26:23.002298 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.002267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b651fb2f-5918-4139-81b6-af25d3c0c151" (UID: "b651fb2f-5918-4139-81b6-af25d3c0c151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:23.002384 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.002261    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" (UID: "9d84bfd5-3699-41ca-81cd-d241b2e9b0e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:23.003997 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.003976    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m" (OuterVolumeSpecName: "kube-api-access-rnm7m") pod "9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" (UID: "9d84bfd5-3699-41ca-81cd-d241b2e9b0e0"). InnerVolumeSpecName "kube-api-access-rnm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:23.004097 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.004073    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz" (OuterVolumeSpecName: "kube-api-access-f4xrz") pod "b651fb2f-5918-4139-81b6-af25d3c0c151" (UID: "b651fb2f-5918-4139-81b6-af25d3c0c151"). InnerVolumeSpecName "kube-api-access-f4xrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:23.102534 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.102503    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xrz\" (UniqueName: \"kubernetes.io/projected/b651fb2f-5918-4139-81b6-af25d3c0c151-kube-api-access-f4xrz\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.102534 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.102530    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnm7m\" (UniqueName: \"kubernetes.io/projected/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-kube-api-access-rnm7m\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.102682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.102539    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.102682 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.102549    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b651fb2f-5918-4139-81b6-af25d3c0c151-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.318122 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.318090    2562 generic.go:356] "Generic (PLEG): container finished" podID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" containerID="d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2" exitCode=0
Sep 10 21:26:23.318322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.318123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" event={"ID":"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0","Type":"ContainerDied","Data":"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"}
Sep 10 21:26:23.318322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.318136    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"
Sep 10 21:26:23.318322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.318170    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4" event={"ID":"9d84bfd5-3699-41ca-81cd-d241b2e9b0e0","Type":"ContainerDied","Data":"c8f09611130e3aa772846603a0b0f6de1251b044a95dbb46a52918f3ef3db9ba"}
Sep 10 21:26:23.318322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.318190    2562 scope.go:117] "RemoveContainer" containerID="d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"
Sep 10 21:26:23.319766 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.319741    2562 generic.go:356] "Generic (PLEG): container finished" podID="b651fb2f-5918-4139-81b6-af25d3c0c151" containerID="e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd" exitCode=0
Sep 10 21:26:23.319889 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.319800    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"
Sep 10 21:26:23.319889 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.319872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" event={"ID":"b651fb2f-5918-4139-81b6-af25d3c0c151","Type":"ContainerDied","Data":"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"}
Sep 10 21:26:23.320023 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.319921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v" event={"ID":"b651fb2f-5918-4139-81b6-af25d3c0c151","Type":"ContainerDied","Data":"203fa7c61e81978f7a9503b15dbda1046397c8204c4eb40d54b5be376bd7231b"}
Sep 10 21:26:23.327762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.327743    2562 scope.go:117] "RemoveContainer" containerID="d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"
Sep 10 21:26:23.328109 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:26:23.328080    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2\": container with ID starting with d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2 not found: ID does not exist" containerID="d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"
Sep 10 21:26:23.328210 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.328119    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2"} err="failed to get container status \"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2\": rpc error: code = NotFound desc = could not find container \"d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2\": container with ID starting with d862367f055c24378d415e6c23070c65e5c473ba9dc01768770a5d69e89a85f2 not found: ID does not exist"
Sep 10 21:26:23.328210 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.328144    2562 scope.go:117] "RemoveContainer" containerID="e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"
Sep 10 21:26:23.336300 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.336276    2562 scope.go:117] "RemoveContainer" containerID="e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"
Sep 10 21:26:23.336594 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:26:23.336571    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd\": container with ID starting with e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd not found: ID does not exist" containerID="e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"
Sep 10 21:26:23.336669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.336600    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd"} err="failed to get container status \"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd\": rpc error: code = NotFound desc = could not find container \"e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd\": container with ID starting with e97b3c332fba51b193e812ebd50edb6847403149bb9f3de400c3b52385fa34dd not found: ID does not exist"
Sep 10 21:26:23.341978 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.341955    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:26:23.348406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.348377    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8623b2-7f6cf69bc8-vsbq4"]
Sep 10 21:26:23.357034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.357014    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:26:23.362437 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:23.362416    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8623b2-766c5bbfb5-zr64v"]
Sep 10 21:26:24.617860 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:24.617827    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" path="/var/lib/kubelet/pods/9d84bfd5-3699-41ca-81cd-d241b2e9b0e0/volumes"
Sep 10 21:26:24.618313 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:24.618189    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b651fb2f-5918-4139-81b6-af25d3c0c151" path="/var/lib/kubelet/pods/b651fb2f-5918-4139-81b6-af25d3c0c151/volumes"
Sep 10 21:26:29.640276 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640244    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"]
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640548    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d674bdf8-00e7-4981-bf33-a2a931b4884e" containerName="main"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640558    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d674bdf8-00e7-4981-bf33-a2a931b4884e" containerName="main"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640573    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640579    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640587    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a616e0-918a-4e92-9fcb-ea8d43126240" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640592    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a616e0-918a-4e92-9fcb-ea8d43126240" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebdaa269-5442-451f-a604-6d3c31d61d87" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640607    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebdaa269-5442-451f-a604-6d3c31d61d87" containerName="maas-api"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640617    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" containerName="payload-pre-processing"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640623    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" containerName="payload-pre-processing"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640629    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b651fb2f-5918-4139-81b6-af25d3c0c151" containerName="payload-processing"
Sep 10 21:26:29.640655 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640634    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b651fb2f-5918-4139-81b6-af25d3c0c151" containerName="payload-processing"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640692    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d84bfd5-3699-41ca-81cd-d241b2e9b0e0" containerName="payload-pre-processing"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640701    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32cbf91-5004-4610-88c9-a1e555c5c8d8" containerName="maas-api"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640706    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebdaa269-5442-451f-a604-6d3c31d61d87" containerName="maas-api"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640711    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b651fb2f-5918-4139-81b6-af25d3c0c151" containerName="payload-processing"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640716    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a616e0-918a-4e92-9fcb-ea8d43126240" containerName="maas-api"
Sep 10 21:26:29.641066 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.640721    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d674bdf8-00e7-4981-bf33-a2a931b4884e" containerName="main"
Sep 10 21:26:29.643708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.643691    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:29.646379 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.646354    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:26:29.646503 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.646395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z6hlm\""
Sep 10 21:26:29.650215 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.650191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"]
Sep 10 21:26:29.755630 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.755598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj\" (UID: \"b3028be0-127a-474d-ab20-669d69119cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:29.856622 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.856582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj\" (UID: \"b3028be0-127a-474d-ab20-669d69119cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:29.857255 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.857234    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj\" (UID: \"b3028be0-127a-474d-ab20-669d69119cf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:29.955774 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:29.955726    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:30.223972 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:30.223946    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"]
Sep 10 21:26:30.226065 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:26:30.226033    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3028be0_127a_474d_ab20_669d69119cf0.slice/crio-50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34 WatchSource:0}: Error finding container 50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34: Status 404 returned error can't find the container with id 50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34
Sep 10 21:26:30.355869 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:30.355835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj" event={"ID":"b3028be0-127a-474d-ab20-669d69119cf0","Type":"ContainerStarted","Data":"50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34"}
Sep 10 21:26:31.362195 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:31.362163    2562 generic.go:356] "Generic (PLEG): container finished" podID="b3028be0-127a-474d-ab20-669d69119cf0" containerID="9f86a5a6ca64c5fc4f811176790c30f58b5747c9d0c151bb823c4b0613bdef28" exitCode=0
Sep 10 21:26:31.362583 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:31.362259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj" event={"ID":"b3028be0-127a-474d-ab20-669d69119cf0","Type":"ContainerDied","Data":"9f86a5a6ca64c5fc4f811176790c30f58b5747c9d0c151bb823c4b0613bdef28"}
Sep 10 21:26:32.489430 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:32.489403    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:32.576357 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:32.576334    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca\") pod \"b3028be0-127a-474d-ab20-669d69119cf0\" (UID: \"b3028be0-127a-474d-ab20-669d69119cf0\") "
Sep 10 21:26:32.576654 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:32.576632    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3028be0-127a-474d-ab20-669d69119cf0" (UID: "b3028be0-127a-474d-ab20-669d69119cf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:32.677405 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:32.677376    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3028be0-127a-474d-ab20-669d69119cf0-maas-api-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:33.372937 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:33.372902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj" event={"ID":"b3028be0-127a-474d-ab20-669d69119cf0","Type":"ContainerDied","Data":"50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34"}
Sep 10 21:26:33.372937 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:33.372931    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"
Sep 10 21:26:33.373150 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:33.372951    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50ea5deabaa5c4d3577495dc3503de96e2345d7d6384b4c993a2b1e821eb2a34"
Sep 10 21:26:40.114017 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.113980    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:26:40.114434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.114210    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" podUID="c42783ce-3f4f-40e1-91d2-3e948ca98314" containerName="payload-pre-processing" containerID="cri-o://71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac" gracePeriod=30
Sep 10 21:26:40.275240 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.275199    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"]
Sep 10 21:26:40.277603 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.277580    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4eb314-nlqsj"]
Sep 10 21:26:40.361924 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.361904    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:26:40.408276 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.408204    2562 generic.go:356] "Generic (PLEG): container finished" podID="c42783ce-3f4f-40e1-91d2-3e948ca98314" containerID="71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac" exitCode=0
Sep 10 21:26:40.408276 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.408243    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"
Sep 10 21:26:40.408475 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.408296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" event={"ID":"c42783ce-3f4f-40e1-91d2-3e948ca98314","Type":"ContainerDied","Data":"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"}
Sep 10 21:26:40.408475 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.408351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd" event={"ID":"c42783ce-3f4f-40e1-91d2-3e948ca98314","Type":"ContainerDied","Data":"f24c1b2bfa8678c582916359c44c559a68beabefa49001dcff40ccbda4900474"}
Sep 10 21:26:40.408475 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.408380    2562 scope.go:117] "RemoveContainer" containerID="71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"
Sep 10 21:26:40.416157 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.416141    2562 scope.go:117] "RemoveContainer" containerID="71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"
Sep 10 21:26:40.416413 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:26:40.416395    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac\": container with ID starting with 71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac not found: ID does not exist" containerID="71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"
Sep 10 21:26:40.416481 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.416419    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac"} err="failed to get container status \"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac\": rpc error: code = NotFound desc = could not find container \"71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac\": container with ID starting with 71db08aa2df2bd751fa1d41573fd0ec2a76a359a055c9df91798b67c8a0c8aac not found: ID does not exist"
Sep 10 21:26:40.436752 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.436733    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l9zq\" (UniqueName: \"kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq\") pod \"c42783ce-3f4f-40e1-91d2-3e948ca98314\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") "
Sep 10 21:26:40.436851 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.436764    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume\") pod \"c42783ce-3f4f-40e1-91d2-3e948ca98314\" (UID: \"c42783ce-3f4f-40e1-91d2-3e948ca98314\") "
Sep 10 21:26:40.437151 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.437128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c42783ce-3f4f-40e1-91d2-3e948ca98314" (UID: "c42783ce-3f4f-40e1-91d2-3e948ca98314"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:40.438759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.438737    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq" (OuterVolumeSpecName: "kube-api-access-4l9zq") pod "c42783ce-3f4f-40e1-91d2-3e948ca98314" (UID: "c42783ce-3f4f-40e1-91d2-3e948ca98314"). InnerVolumeSpecName "kube-api-access-4l9zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:40.538310 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.538279    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9zq\" (UniqueName: \"kubernetes.io/projected/c42783ce-3f4f-40e1-91d2-3e948ca98314-kube-api-access-4l9zq\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:40.538310 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.538308    2562 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42783ce-3f4f-40e1-91d2-3e948ca98314-plugins-config-volume\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:26:40.618298 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.618267    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3028be0-127a-474d-ab20-669d69119cf0" path="/var/lib/kubelet/pods/b3028be0-127a-474d-ab20-669d69119cf0/volumes"
Sep 10 21:26:40.722539 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.722512    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:26:40.726054 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:40.726032    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4eb314-776dd4d98-ng9bd"]
Sep 10 21:26:42.617877 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:42.617847    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42783ce-3f4f-40e1-91d2-3e948ca98314" path="/var/lib/kubelet/pods/c42783ce-3f4f-40e1-91d2-3e948ca98314/volumes"
Sep 10 21:26:52.743608 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:52.743578    2562 scope.go:117] "RemoveContainer" containerID="ea574253fcccde2d5422053305bd996adc45da4fd2c76a76a684128ac1589299"
Sep 10 21:26:52.751283 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:26:52.751262    2562 scope.go:117] "RemoveContainer" containerID="16b6c5c4f3ef4eb20617e8be94ab6582aacdde056c7fae4f2aa72f579a717479"
Sep 10 21:27:52.783467 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:27:52.783432    2562 scope.go:117] "RemoveContainer" containerID="ddd6a7667d6fb01a730dcc725a4c5ff801076f85e8784c20b15e5c2407669e0c"
Sep 10 21:27:52.791281 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:27:52.791259    2562 scope.go:117] "RemoveContainer" containerID="bc3664cb6fd6f0eceaabb507ce23c2448d688715562c7638ef06815a3a425f1a"
Sep 10 21:27:52.798351 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:27:52.798327    2562 scope.go:117] "RemoveContainer" containerID="795332b02a56920e9a3bf9cd9f02ec8a7bad7b40015bd814e602f922bb428cc5"
Sep 10 21:30:59.351994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.351922    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352230    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42783ce-3f4f-40e1-91d2-3e948ca98314" containerName="payload-pre-processing"
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352243    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42783ce-3f4f-40e1-91d2-3e948ca98314" containerName="payload-pre-processing"
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352256    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3028be0-127a-474d-ab20-669d69119cf0" containerName="revoke-keys"
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3028be0-127a-474d-ab20-669d69119cf0" containerName="revoke-keys"
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352324    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42783ce-3f4f-40e1-91d2-3e948ca98314" containerName="payload-pre-processing"
Sep 10 21:30:59.352415 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.352331    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3028be0-127a-474d-ab20-669d69119cf0" containerName="revoke-keys"
Sep 10 21:30:59.355102 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.355085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.357771 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.357750    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a21ca76f-kserve-self-signed-certs\""
Sep 10 21:30:59.365569 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.365545    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:30:59.464145 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.464309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464163    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.464309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464210    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.464309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6bdt\" (UniqueName: \"kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.464309 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.464462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.464322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565262 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565262 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6bdt\" (UniqueName: \"kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565499 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565705 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565667    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565764 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565722    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.565850 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.565830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.567577 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.567556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.567994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.567971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.572876 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.572857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6bdt\" (UniqueName: \"kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt\") pod \"e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") " pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.664434 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.664382    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:30:59.932025 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:30:59.932002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:30:59.934275 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:30:59.934239    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14b2146_32e6_4695_bf99_083740e24382.slice/crio-03dd5f0a7e3f7dbb8ee436e29229a5b8284a336b6ec28e1d1dadbec2803a5ec0 WatchSource:0}: Error finding container 03dd5f0a7e3f7dbb8ee436e29229a5b8284a336b6ec28e1d1dadbec2803a5ec0: Status 404 returned error can't find the container with id 03dd5f0a7e3f7dbb8ee436e29229a5b8284a336b6ec28e1d1dadbec2803a5ec0
Sep 10 21:31:00.513612 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:00.513576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" event={"ID":"b14b2146-32e6-4695-bf99-083740e24382","Type":"ContainerStarted","Data":"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"}
Sep 10 21:31:00.514016 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:00.513620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" event={"ID":"b14b2146-32e6-4695-bf99-083740e24382","Type":"ContainerStarted","Data":"03dd5f0a7e3f7dbb8ee436e29229a5b8284a336b6ec28e1d1dadbec2803a5ec0"}
Sep 10 21:31:00.530674 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:00.530628    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" podStartSLOduration=1.530613839 podStartE2EDuration="1.530613839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:31:00.529493708 +0000 UTC m=+1988.450114095" watchObservedRunningTime="2026-09-10 21:31:00.530613839 +0000 UTC m=+1988.451234225"
Sep 10 21:31:05.654028 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.653990    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:05.657652 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.657634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.660241 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.660222    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-a21ca76f-kserve-self-signed-certs\""
Sep 10 21:31:05.666511 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.666485    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:05.714462 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.714591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714469    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.714591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k274k\" (UniqueName: \"kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.714669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.714669 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.714734 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.714682    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815104 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k274k\" (UniqueName: \"kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815233 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815233 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815233 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815233 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815613 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815589    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.815666 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.815621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.817566 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.817548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.817640 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.817615    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.822987 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.822967    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k274k\" (UniqueName: \"kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k\") pod \"e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") " pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:05.967946 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:05.967896    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:06.237927 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:06.237904    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:06.241665 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:31:06.241632    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc836082e_030f_41ad_b15a_fd5f89cf655b.slice/crio-3562b02e7acbaa57d675cd3e6e2da82311dc75389f779903b850dfc35d0c6a52 WatchSource:0}: Error finding container 3562b02e7acbaa57d675cd3e6e2da82311dc75389f779903b850dfc35d0c6a52: Status 404 returned error can't find the container with id 3562b02e7acbaa57d675cd3e6e2da82311dc75389f779903b850dfc35d0c6a52
Sep 10 21:31:06.538133 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:06.538052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" event={"ID":"c836082e-030f-41ad-b15a-fd5f89cf655b","Type":"ContainerStarted","Data":"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"}
Sep 10 21:31:06.538133 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:06.538098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" event={"ID":"c836082e-030f-41ad-b15a-fd5f89cf655b","Type":"ContainerStarted","Data":"3562b02e7acbaa57d675cd3e6e2da82311dc75389f779903b850dfc35d0c6a52"}
Sep 10 21:31:06.554178 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:06.554133    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" podStartSLOduration=1.554119768 podStartE2EDuration="1.554119768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:31:06.553846265 +0000 UTC m=+1994.474466678" watchObservedRunningTime="2026-09-10 21:31:06.554119768 +0000 UTC m=+1994.474740154"
Sep 10 21:31:09.664893 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:09.664855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:09.664893 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:09.664902    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:09.677585 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:09.677555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:10.567730 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:10.567699    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:15.968960 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:15.968923    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:15.969326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:15.969089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:15.981621 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:15.981600    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:16.594051 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:16.594025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:36.439471 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:36.439436    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:31:37.341611 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:37.341576    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:51.459450 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.459386    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" podUID="b14b2146-32e6-4695-bf99-083740e24382" containerName="main" containerID="cri-o://2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c" gracePeriod=45
Sep 10 21:31:51.700170 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.700147    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:51.732336 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.732273    2562 generic.go:356] "Generic (PLEG): container finished" podID="b14b2146-32e6-4695-bf99-083740e24382" containerID="2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c" exitCode=0
Sep 10 21:31:51.732336 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.732314    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"
Sep 10 21:31:51.732524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.732367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" event={"ID":"b14b2146-32e6-4695-bf99-083740e24382","Type":"ContainerDied","Data":"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"}
Sep 10 21:31:51.732524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.732415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm" event={"ID":"b14b2146-32e6-4695-bf99-083740e24382","Type":"ContainerDied","Data":"03dd5f0a7e3f7dbb8ee436e29229a5b8284a336b6ec28e1d1dadbec2803a5ec0"}
Sep 10 21:31:51.732524 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.732437    2562 scope.go:117] "RemoveContainer" containerID="2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"
Sep 10 21:31:51.740526 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.740508    2562 scope.go:117] "RemoveContainer" containerID="2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"
Sep 10 21:31:51.740776 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:31:51.740756    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c\": container with ID starting with 2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c not found: ID does not exist" containerID="2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"
Sep 10 21:31:51.740865 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.740784    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c"} err="failed to get container status \"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c\": rpc error: code = NotFound desc = could not find container \"2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c\": container with ID starting with 2a30b57369d341876b2ca85c9338f08c8361e32d42838c0955c394686541c52c not found: ID does not exist"
Sep 10 21:31:51.782231 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6bdt\" (UniqueName: \"kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782344 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782282    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782344 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782354    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782388    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs\") pod \"b14b2146-32e6-4695-bf99-083740e24382\" (UID: \"b14b2146-32e6-4695-bf99-083740e24382\") "
Sep 10 21:31:51.782597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782572    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:51.782597 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home" (OuterVolumeSpecName: "home") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:51.782733 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.782626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache" (OuterVolumeSpecName: "model-cache") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:51.784355 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.784330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm" (OuterVolumeSpecName: "dshm") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:51.784453 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.784350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt" (OuterVolumeSpecName: "kube-api-access-s6bdt") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "kube-api-access-s6bdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:31:51.784453 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.784392    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b14b2146-32e6-4695-bf99-083740e24382" (UID: "b14b2146-32e6-4695-bf99-083740e24382"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:31:51.883826 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883777    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:51.883826 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883824    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:51.883994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883840    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:51.883994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883853    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14b2146-32e6-4695-bf99-083740e24382-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:51.883994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6bdt\" (UniqueName: \"kubernetes.io/projected/b14b2146-32e6-4695-bf99-083740e24382-kube-api-access-s6bdt\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:51.883994 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:51.883877    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14b2146-32e6-4695-bf99-083740e24382-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.060074 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.060047    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:31:52.063537 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.063513    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a21ca76f-kserve-85f6567895-nv9nm"]
Sep 10 21:31:52.360720 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.360606    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" podUID="c836082e-030f-41ad-b15a-fd5f89cf655b" containerName="main" containerID="cri-o://1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686" gracePeriod=45
Sep 10 21:31:52.603740 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.603713    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:52.618591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.618526    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b14b2146-32e6-4695-bf99-083740e24382" path="/var/lib/kubelet/pods/b14b2146-32e6-4695-bf99-083740e24382/volumes"
Sep 10 21:31:52.690323 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690290    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k274k\" (UniqueName: \"kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690447 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690447 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690447 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690385    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690612 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690455    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690612 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir\") pod \"c836082e-030f-41ad-b15a-fd5f89cf655b\" (UID: \"c836082e-030f-41ad-b15a-fd5f89cf655b\") "
Sep 10 21:31:52.690612 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home" (OuterVolumeSpecName: "home") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:52.690762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690749    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-home\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.690831 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:52.690831 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.690782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:52.692396 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.692376    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k" (OuterVolumeSpecName: "kube-api-access-k274k") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "kube-api-access-k274k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:31:52.692466 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.692408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm" (OuterVolumeSpecName: "dshm") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:52.692466 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.692447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c836082e-030f-41ad-b15a-fd5f89cf655b" (UID: "c836082e-030f-41ad-b15a-fd5f89cf655b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:31:52.738305 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.738275    2562 generic.go:356] "Generic (PLEG): container finished" podID="c836082e-030f-41ad-b15a-fd5f89cf655b" containerID="1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686" exitCode=0
Sep 10 21:31:52.738408 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.738319    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"
Sep 10 21:31:52.738408 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.738332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" event={"ID":"c836082e-030f-41ad-b15a-fd5f89cf655b","Type":"ContainerDied","Data":"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"}
Sep 10 21:31:52.738408 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.738362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5" event={"ID":"c836082e-030f-41ad-b15a-fd5f89cf655b","Type":"ContainerDied","Data":"3562b02e7acbaa57d675cd3e6e2da82311dc75389f779903b850dfc35d0c6a52"}
Sep 10 21:31:52.738408 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.738378    2562 scope.go:117] "RemoveContainer" containerID="1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"
Sep 10 21:31:52.746908 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.746891    2562 scope.go:117] "RemoveContainer" containerID="1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"
Sep 10 21:31:52.747216 ip-10-0-135-61 kubenswrapper[2562]: E0910 21:31:52.747197    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686\": container with ID starting with 1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686 not found: ID does not exist" containerID="1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"
Sep 10 21:31:52.747262 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.747227    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686"} err="failed to get container status \"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686\": rpc error: code = NotFound desc = could not find container \"1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686\": container with ID starting with 1ead1d2cd971d3b5f72625ab3f059c34fc4beea559a9236ec8f1a1e605d91686 not found: ID does not exist"
Sep 10 21:31:52.760705 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.760684    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:52.763753 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.763732    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-a21ca76f-kserve-5444d786c-87gg5"]
Sep 10 21:31:52.791958 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.791936    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c836082e-030f-41ad-b15a-fd5f89cf655b-tls-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.791958 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.791958    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-model-cache\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.792085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.791971    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-tmp-dir\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.792085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.791984    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k274k\" (UniqueName: \"kubernetes.io/projected/c836082e-030f-41ad-b15a-fd5f89cf655b-kube-api-access-k274k\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:52.792085 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:52.791997    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c836082e-030f-41ad-b15a-fd5f89cf655b-dshm\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:31:54.049034 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.048996    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:31:54.049486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049398    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c836082e-030f-41ad-b15a-fd5f89cf655b" containerName="main"
Sep 10 21:31:54.049486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049416    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c836082e-030f-41ad-b15a-fd5f89cf655b" containerName="main"
Sep 10 21:31:54.049486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049434    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b14b2146-32e6-4695-bf99-083740e24382" containerName="main"
Sep 10 21:31:54.049486 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049442    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b14b2146-32e6-4695-bf99-083740e24382" containerName="main"
Sep 10 21:31:54.049699 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049524    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b14b2146-32e6-4695-bf99-083740e24382" containerName="main"
Sep 10 21:31:54.049699 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.049538    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c836082e-030f-41ad-b15a-fd5f89cf655b" containerName="main"
Sep 10 21:31:54.053900 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.053877    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.056611 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.056589    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:31:54.057755 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.057736    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:31:54.057883 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.057764    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:31:54.057883 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.057854    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tww8l\""
Sep 10 21:31:54.058031 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.058018    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:31:54.062502 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.062381    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:31:54.101456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.101419    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.101456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.101457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.101759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.101525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.101759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.101578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrnmv\" (UniqueName: \"kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.202302 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.202267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrnmv\" (UniqueName: \"kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.202442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.202320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.202442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.202368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.202442 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.202433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.203040 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.203019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.204747 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.204725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.204844 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.204770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.209556 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.209538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrnmv\" (UniqueName: \"kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv\") pod \"maas-api-d8489f74-qvccg\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") " pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.366972 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.366887    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:54.618821 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.618727    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c836082e-030f-41ad-b15a-fd5f89cf655b" path="/var/lib/kubelet/pods/c836082e-030f-41ad-b15a-fd5f89cf655b/volumes"
Sep 10 21:31:54.635713 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.635685    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:31:54.639081 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:31:54.639043    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee3fab8_24bd_43ef_b752_e96a6dd90246.slice/crio-07320d449ad48b4f496c634d70eff932f1b1d5ce95f4c5091bb2097880c518e2 WatchSource:0}: Error finding container 07320d449ad48b4f496c634d70eff932f1b1d5ce95f4c5091bb2097880c518e2: Status 404 returned error can't find the container with id 07320d449ad48b4f496c634d70eff932f1b1d5ce95f4c5091bb2097880c518e2
Sep 10 21:31:54.643341 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.642724    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:31:54.748049 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:54.748016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" event={"ID":"aee3fab8-24bd-43ef-b752-e96a6dd90246","Type":"ContainerStarted","Data":"07320d449ad48b4f496c634d70eff932f1b1d5ce95f4c5091bb2097880c518e2"}
Sep 10 21:31:55.753855 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:55.753817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" event={"ID":"aee3fab8-24bd-43ef-b752-e96a6dd90246","Type":"ContainerStarted","Data":"eecff20e147b97213f8e7372664f35042b17a7a69147ccf1d9b685b38e636fc7"}
Sep 10 21:31:55.754244 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:55.753901    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:31:55.772548 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:31:55.772503    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" podStartSLOduration=1.356643327 podStartE2EDuration="1.772489064s" totalImagesPullingTime="415.845737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:31:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:31:55.770355309 +0000 UTC m=+2043.690975705" watchObservedRunningTime="2026-09-10 21:31:55.772489064 +0000 UTC m=+2043.693109450"
Sep 10 21:32:01.589801 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.589761    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:32:01.590188 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.590062    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" podUID="aee3fab8-24bd-43ef-b752-e96a6dd90246" containerName="maas-api" containerID="cri-o://eecff20e147b97213f8e7372664f35042b17a7a69147ccf1d9b685b38e636fc7" gracePeriod=30
Sep 10 21:32:01.594946 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.594924    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:32:01.623863 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.623833    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"]
Sep 10 21:32:01.626728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.626711    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.635363 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.635340    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"]
Sep 10 21:32:01.767431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.767396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-maas-api-tls\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.767431 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.767441    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed31a5-dd69-4578-a482-251be987b81c-openshift-service-ca\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.767663 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.767498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnz7z\" (UniqueName: \"kubernetes.io/projected/91ed31a5-dd69-4578-a482-251be987b81c-kube-api-access-jnz7z\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.767663 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.767540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-metrics-certs\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.780382 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.780351    2562 generic.go:356] "Generic (PLEG): container finished" podID="aee3fab8-24bd-43ef-b752-e96a6dd90246" containerID="eecff20e147b97213f8e7372664f35042b17a7a69147ccf1d9b685b38e636fc7" exitCode=0
Sep 10 21:32:01.780528 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.780378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" event={"ID":"aee3fab8-24bd-43ef-b752-e96a6dd90246","Type":"ContainerDied","Data":"eecff20e147b97213f8e7372664f35042b17a7a69147ccf1d9b685b38e636fc7"}
Sep 10 21:32:01.829662 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.829640    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:32:01.868370 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.868307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-maas-api-tls\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.868370 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.868354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed31a5-dd69-4578-a482-251be987b81c-openshift-service-ca\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.868571 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.868377    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnz7z\" (UniqueName: \"kubernetes.io/projected/91ed31a5-dd69-4578-a482-251be987b81c-kube-api-access-jnz7z\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.868571 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.868396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-metrics-certs\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.869064 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.869043    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ed31a5-dd69-4578-a482-251be987b81c-openshift-service-ca\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.870670 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.870650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-maas-api-tls\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.870759 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.870711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91ed31a5-dd69-4578-a482-251be987b81c-metrics-certs\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.877492 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.877473    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnz7z\" (UniqueName: \"kubernetes.io/projected/91ed31a5-dd69-4578-a482-251be987b81c-kube-api-access-jnz7z\") pod \"maas-api-58d8fbff5d-rvgmf\" (UID: \"91ed31a5-dd69-4578-a482-251be987b81c\") " pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.962262 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.962227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:01.969187 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.969169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs\") pod \"aee3fab8-24bd-43ef-b752-e96a6dd90246\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") "
Sep 10 21:32:01.969254 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.969239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrnmv\" (UniqueName: \"kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv\") pod \"aee3fab8-24bd-43ef-b752-e96a6dd90246\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") "
Sep 10 21:32:01.969325 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.969312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls\") pod \"aee3fab8-24bd-43ef-b752-e96a6dd90246\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") "
Sep 10 21:32:01.969379 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.969365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca\") pod \"aee3fab8-24bd-43ef-b752-e96a6dd90246\" (UID: \"aee3fab8-24bd-43ef-b752-e96a6dd90246\") "
Sep 10 21:32:01.969788 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.969756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aee3fab8-24bd-43ef-b752-e96a6dd90246" (UID: "aee3fab8-24bd-43ef-b752-e96a6dd90246"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:32:01.971203 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.971179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aee3fab8-24bd-43ef-b752-e96a6dd90246" (UID: "aee3fab8-24bd-43ef-b752-e96a6dd90246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:32:01.971326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.971305    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv" (OuterVolumeSpecName: "kube-api-access-zrnmv") pod "aee3fab8-24bd-43ef-b752-e96a6dd90246" (UID: "aee3fab8-24bd-43ef-b752-e96a6dd90246"). InnerVolumeSpecName "kube-api-access-zrnmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:32:01.971326 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:01.971317    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aee3fab8-24bd-43ef-b752-e96a6dd90246" (UID: "aee3fab8-24bd-43ef-b752-e96a6dd90246"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:32:02.070762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.070731    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee3fab8-24bd-43ef-b752-e96a6dd90246-openshift-service-ca\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:32:02.070762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.070758    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-metrics-certs\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:32:02.070762 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.070767    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrnmv\" (UniqueName: \"kubernetes.io/projected/aee3fab8-24bd-43ef-b752-e96a6dd90246-kube-api-access-zrnmv\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:32:02.070974 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.070775    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aee3fab8-24bd-43ef-b752-e96a6dd90246-maas-api-tls\") on node \"ip-10-0-135-61.ec2.internal\" DevicePath \"\""
Sep 10 21:32:02.230764 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.230744    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"]
Sep 10 21:32:02.233194 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:32:02.233165    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ed31a5_dd69_4578_a482_251be987b81c.slice/crio-8b555d47c270a91f6eb35e8dbc4ec94f7c54b2ad6b9023638c679d15eff67deb WatchSource:0}: Error finding container 8b555d47c270a91f6eb35e8dbc4ec94f7c54b2ad6b9023638c679d15eff67deb: Status 404 returned error can't find the container with id 8b555d47c270a91f6eb35e8dbc4ec94f7c54b2ad6b9023638c679d15eff67deb
Sep 10 21:32:02.786716 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.786679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf" event={"ID":"91ed31a5-dd69-4578-a482-251be987b81c","Type":"ContainerStarted","Data":"4f939f7af63e4bff6eedbc5be7b06d451ebe6dda8e88494247e5f0a51ae49ed0"}
Sep 10 21:32:02.787121 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.786744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf" event={"ID":"91ed31a5-dd69-4578-a482-251be987b81c","Type":"ContainerStarted","Data":"8b555d47c270a91f6eb35e8dbc4ec94f7c54b2ad6b9023638c679d15eff67deb"}
Sep 10 21:32:02.787121 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.786767    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:02.788183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.788165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg" event={"ID":"aee3fab8-24bd-43ef-b752-e96a6dd90246","Type":"ContainerDied","Data":"07320d449ad48b4f496c634d70eff932f1b1d5ce95f4c5091bb2097880c518e2"}
Sep 10 21:32:02.788287 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.788190    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-d8489f74-qvccg"
Sep 10 21:32:02.788287 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.788205    2562 scope.go:117] "RemoveContainer" containerID="eecff20e147b97213f8e7372664f35042b17a7a69147ccf1d9b685b38e636fc7"
Sep 10 21:32:02.804040 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.803995    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf" podStartSLOduration=1.335883883 podStartE2EDuration="1.803978345s" totalImagesPullingTime="468.094462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:02.803188763 +0000 UTC m=+2050.723809171" watchObservedRunningTime="2026-09-10 21:32:02.803978345 +0000 UTC m=+2050.724598739"
Sep 10 21:32:02.817289 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.817262    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:32:02.822564 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:02.822541    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-d8489f74-qvccg"]
Sep 10 21:32:04.617577 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:04.617544    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee3fab8-24bd-43ef-b752-e96a6dd90246" path="/var/lib/kubelet/pods/aee3fab8-24bd-43ef-b752-e96a6dd90246/volumes"
Sep 10 21:32:08.799167 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:08.799135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d8fbff5d-rvgmf"
Sep 10 21:32:23.644642 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:23.644539    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b58cd5757-8dmv6_7b104a17-3c4f-4bfc-a5fc-cc8b8989b886/authorino/0.log"
Sep 10 21:32:29.486407 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:29.486373    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d855cccc6-nln6b_1c5265dc-75d5-430b-934b-91c6572aebf3/copy-manifests/0.log"
Sep 10 21:32:29.499286 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:29.499258    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d855cccc6-nln6b_1c5265dc-75d5-430b-934b-91c6572aebf3/manager/0.log"
Sep 10 21:32:29.816929 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:29.816850    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58d8fbff5d-rvgmf_91ed31a5-dd69-4578-a482-251be987b81c/maas-api/0.log"
Sep 10 21:32:30.030688 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:30.030658    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817915-vr9g7_fcb11260-d76f-4e08-8354-06bd58501f62/cleanup/0.log"
Sep 10 21:32:30.975824 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:30.975780    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b58cd5757-8dmv6_7b104a17-3c4f-4bfc-a5fc-cc8b8989b886/authorino/0.log"
Sep 10 21:32:31.302510 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:31.302436    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-6zbkc_ce6f5883-8f55-4aa0-bb32-0b2ec8adea32/kuadrant-console-plugin/0.log"
Sep 10 21:32:32.357479 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:32.357438    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767744946f-t79lq_20403f47-03f1-418b-88e6-24884383c103/kube-auth-proxy/0.log"
Sep 10 21:32:32.686498 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:32.686405    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:32:33.231457 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:33.231432    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lmc2c_86168c85-b00f-4b80-adc8-839c0239a99a/main/0.log"
Sep 10 21:32:33.440354 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:33.440320    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vbltj_83dc4926-909e-4df2-9489-5803dd297c1d/main/0.log"
Sep 10 21:32:33.752756 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:33.752725    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7ghwk_7fd01427-bb09-430d-a4ed-055927b653d1/main/0.log"
Sep 10 21:32:39.045563 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.045525    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47z52/must-gather-zlzc8"]
Sep 10 21:32:39.046183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.045968    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee3fab8-24bd-43ef-b752-e96a6dd90246" containerName="maas-api"
Sep 10 21:32:39.046183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.045990    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee3fab8-24bd-43ef-b752-e96a6dd90246" containerName="maas-api"
Sep 10 21:32:39.046183 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.046076    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee3fab8-24bd-43ef-b752-e96a6dd90246" containerName="maas-api"
Sep 10 21:32:39.049573 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.049552    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.052089 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.052063    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47z52\"/\"openshift-service-ca.crt\""
Sep 10 21:32:39.053271 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.053250    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47z52\"/\"default-dockercfg-fsfx2\""
Sep 10 21:32:39.053377 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.053282    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47z52\"/\"kube-root-ca.crt\""
Sep 10 21:32:39.057043 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.057022    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47z52/must-gather-zlzc8"]
Sep 10 21:32:39.063322 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.063302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169a9d64-a3b7-41e3-9b21-33931102943e-must-gather-output\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.063422 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.063347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw4k\" (UniqueName: \"kubernetes.io/projected/169a9d64-a3b7-41e3-9b21-33931102943e-kube-api-access-ggw4k\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.164380 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.164347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw4k\" (UniqueName: \"kubernetes.io/projected/169a9d64-a3b7-41e3-9b21-33931102943e-kube-api-access-ggw4k\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.164545 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.164465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169a9d64-a3b7-41e3-9b21-33931102943e-must-gather-output\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.164864 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.164843    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/169a9d64-a3b7-41e3-9b21-33931102943e-must-gather-output\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.172264 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.172241    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw4k\" (UniqueName: \"kubernetes.io/projected/169a9d64-a3b7-41e3-9b21-33931102943e-kube-api-access-ggw4k\") pod \"must-gather-zlzc8\" (UID: \"169a9d64-a3b7-41e3-9b21-33931102943e\") " pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.359995 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.359905    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47z52/must-gather-zlzc8"
Sep 10 21:32:39.626361 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.626289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47z52/must-gather-zlzc8"]
Sep 10 21:32:39.628654 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:32:39.628621    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod169a9d64_a3b7_41e3_9b21_33931102943e.slice/crio-4118bc42f7e70caef254e775efba5fd1fc5fc210ff67099602ad569fd04078b8 WatchSource:0}: Error finding container 4118bc42f7e70caef254e775efba5fd1fc5fc210ff67099602ad569fd04078b8: Status 404 returned error can't find the container with id 4118bc42f7e70caef254e775efba5fd1fc5fc210ff67099602ad569fd04078b8
Sep 10 21:32:39.942184 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:39.942151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47z52/must-gather-zlzc8" event={"ID":"169a9d64-a3b7-41e3-9b21-33931102943e","Type":"ContainerStarted","Data":"4118bc42f7e70caef254e775efba5fd1fc5fc210ff67099602ad569fd04078b8"}
Sep 10 21:32:40.951196 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:40.951159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47z52/must-gather-zlzc8" event={"ID":"169a9d64-a3b7-41e3-9b21-33931102943e","Type":"ContainerStarted","Data":"e5589d990336e82986f584e71e020447d80eaa2ee4f2fd60c79a0abb993f29b3"}
Sep 10 21:32:40.951724 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:40.951691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47z52/must-gather-zlzc8" event={"ID":"169a9d64-a3b7-41e3-9b21-33931102943e","Type":"ContainerStarted","Data":"979cc779047d9e38903926079d71167cd4b1cd4bc67d1a4394b59affa5e13e46"}
Sep 10 21:32:40.968252 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:40.968208    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-47z52/must-gather-zlzc8" podStartSLOduration=1.336468075 podStartE2EDuration="1.968192732s" totalImagesPullingTime="631.724657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:40.968155164 +0000 UTC m=+2088.888775550" watchObservedRunningTime="2026-09-10 21:32:40.968192732 +0000 UTC m=+2088.888813115"
Sep 10 21:32:41.764474 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:41.764445    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cvk8k_eca93feb-5081-4d12-ba13-def3a077f69c/global-pull-secret-syncer/0.log"
Sep 10 21:32:41.954282 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:41.954245    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vc8dg_886749a3-399a-4a72-b975-05124965b591/konnectivity-agent/0.log"
Sep 10 21:32:41.995314 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:41.995286    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-61.ec2.internal_7ed6d91cc61f87a0eb8bab0854fc763b/haproxy/0.log"
Sep 10 21:32:45.943827 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:45.943244    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b58cd5757-8dmv6_7b104a17-3c4f-4bfc-a5fc-cc8b8989b886/authorino/0.log"
Sep 10 21:32:46.036266 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:46.036228    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-6zbkc_ce6f5883-8f55-4aa0-bb32-0b2ec8adea32/kuadrant-console-plugin/0.log"
Sep 10 21:32:48.130515 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:48.130471    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzbc_ccfac512-7196-4a29-adeb-1ceef41c63c7/node-exporter/0.log"
Sep 10 21:32:48.150535 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:48.150507    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzbc_ccfac512-7196-4a29-adeb-1ceef41c63c7/kube-rbac-proxy/0.log"
Sep 10 21:32:48.170120 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:48.170020    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5pzbc_ccfac512-7196-4a29-adeb-1ceef41c63c7/init-textfile/0.log"
Sep 10 21:32:51.833283 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:51.833251    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-868fdd4ff8-x88gh_7c62f27b-9878-4335-b986-8bcf7f84a9ff/console/0.log"
Sep 10 21:32:52.167686 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.167606    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"]
Sep 10 21:32:52.174451 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.174425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.180464 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.180428    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"]
Sep 10 21:32:52.305531 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.305485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-proc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.305728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.305555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-sys\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.305728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.305601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-podres\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.305728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.305639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprxc\" (UniqueName: \"kubernetes.io/projected/ee072602-33be-4185-8af1-0defe6fbdc4e-kube-api-access-dprxc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.305728 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.305697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-lib-modules\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.407090 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.407033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprxc\" (UniqueName: \"kubernetes.io/projected/ee072602-33be-4185-8af1-0defe6fbdc4e-kube-api-access-dprxc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.407406 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.407386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-lib-modules\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.407711 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.407691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-proc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.407986 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.407960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-sys\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.408233 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.408214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-podres\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.408383 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.408253    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-sys\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.408590 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.408368    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-lib-modules\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.408714 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.408435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-podres\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.408921 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.408474    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ee072602-33be-4185-8af1-0defe6fbdc4e-proc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.416778 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.416751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprxc\" (UniqueName: \"kubernetes.io/projected/ee072602-33be-4185-8af1-0defe6fbdc4e-kube-api-access-dprxc\") pod \"perf-node-gather-daemonset-j6wqc\" (UID: \"ee072602-33be-4185-8af1-0defe6fbdc4e\") " pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.491335 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.491296    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:52.788164 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.788137    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"]
Sep 10 21:32:52.791615 ip-10-0-135-61 kubenswrapper[2562]: W0910 21:32:52.791582    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee072602_33be_4185_8af1_0defe6fbdc4e.slice/crio-8f7e30b42efb767f23772f487bcd1d2b81c1190711e06694b40d6cd6624985b1 WatchSource:0}: Error finding container 8f7e30b42efb767f23772f487bcd1d2b81c1190711e06694b40d6cd6624985b1: Status 404 returned error can't find the container with id 8f7e30b42efb767f23772f487bcd1d2b81c1190711e06694b40d6cd6624985b1
Sep 10 21:32:52.853180 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:52.853147    2562 scope.go:117] "RemoveContainer" containerID="9f86a5a6ca64c5fc4f811176790c30f58b5747c9d0c151bb823c4b0613bdef28"
Sep 10 21:32:53.028520 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.028466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc" event={"ID":"ee072602-33be-4185-8af1-0defe6fbdc4e","Type":"ContainerStarted","Data":"6507a314fc4f84789ff12d9dc22d51927494f0d0b95f99d1f86c05ced25d3745"}
Sep 10 21:32:53.028708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.028597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc" event={"ID":"ee072602-33be-4185-8af1-0defe6fbdc4e","Type":"ContainerStarted","Data":"8f7e30b42efb767f23772f487bcd1d2b81c1190711e06694b40d6cd6624985b1"}
Sep 10 21:32:53.028708 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.028632    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:53.046421 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.046319    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc" podStartSLOduration=1.046292668 podStartE2EDuration="1.046292668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:53.043538589 +0000 UTC m=+2100.964158996" watchObservedRunningTime="2026-09-10 21:32:53.046292668 +0000 UTC m=+2100.966913055"
Sep 10 21:32:53.815878 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.815850    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q996r_2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a/dns/0.log"
Sep 10 21:32:53.833423 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.833398    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q996r_2f0342b9-8b2d-4baf-b70f-c71bf72b8f7a/kube-rbac-proxy/0.log"
Sep 10 21:32:53.875729 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:53.875695    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hdmm9_a2e81c4a-6431-443e-b0ae-2666493afbd2/dns-node-resolver/0.log"
Sep 10 21:32:54.449690 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:54.449661    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zp29k_2df000b5-6b2e-42f9-b1ab-8ac8ff03da1f/node-ca/0.log"
Sep 10 21:32:55.567536 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:55.567511    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767744946f-t79lq_20403f47-03f1-418b-88e6-24884383c103/kube-auth-proxy/0.log"
Sep 10 21:32:55.673257 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:55.673222    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:32:56.542973 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:56.542944    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8sx7m_f18c590a-d095-44c2-b581-69fb5f7f00cf/serve-healthcheck-canary/0.log"
Sep 10 21:32:57.163088 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:57.163057    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tt76q_69441a4d-87bf-4778-a131-f1058b1d0163/kube-rbac-proxy/0.log"
Sep 10 21:32:57.182203 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:57.182177    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tt76q_69441a4d-87bf-4778-a131-f1058b1d0163/exporter/0.log"
Sep 10 21:32:57.201087 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:57.201053    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tt76q_69441a4d-87bf-4778-a131-f1058b1d0163/extractor/0.log"
Sep 10 21:32:59.044449 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:59.044421    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47z52/perf-node-gather-daemonset-j6wqc"
Sep 10 21:32:59.712948 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:59.712912    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d855cccc6-nln6b_1c5265dc-75d5-430b-934b-91c6572aebf3/manager/0.log"
Sep 10 21:32:59.727043 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:32:59.727021    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d855cccc6-nln6b_1c5265dc-75d5-430b-934b-91c6572aebf3/copy-manifests/0.log"
Sep 10 21:33:00.976710 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:00.976677    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f8c8876d7-jxlnl_708dbef9-0a27-4230-a958-a73588f15671/manager/0.log"
Sep 10 21:33:02.580360 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:02.580332    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767744946f-t79lq_20403f47-03f1-418b-88e6-24884383c103/kube-auth-proxy/0.log"
Sep 10 21:33:02.653797 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:02.653762    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:33:04.476502 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:04.476473    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767744946f-t79lq_20403f47-03f1-418b-88e6-24884383c103/kube-auth-proxy/0.log"
Sep 10 21:33:04.559423 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:04.559387    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-6fs7j_3a68c30c-63f3-45c1-b656-976e1710d528/payload-processing/0.log"
Sep 10 21:33:07.893191 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.893156    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:33:07.912200 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.912166    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/egress-router-binary-copy/0.log"
Sep 10 21:33:07.933273 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.933249    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/cni-plugins/0.log"
Sep 10 21:33:07.952266 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.952237    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/bond-cni-plugin/0.log"
Sep 10 21:33:07.971591 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.971553    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/routeoverride-cni/0.log"
Sep 10 21:33:07.989047 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:07.989025    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/whereabouts-cni-bincopy/0.log"
Sep 10 21:33:08.007212 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:08.007182    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s64mf_4bd50d37-f232-4a31-bd09-b30164a06d08/whereabouts-cni/0.log"
Sep 10 21:33:08.101456 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:08.101417    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jjdbq_ca3b6042-30a3-489f-a16c-ac57565916c1/kube-multus/0.log"
Sep 10 21:33:08.210737 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:08.210706    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xrp9_d82ca890-8cb0-42e1-9325-555f07de811d/network-metrics-daemon/0.log"
Sep 10 21:33:08.225663 ip-10-0-135-61 kubenswrapper[2562]: I0910 21:33:08.225642    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9xrp9_d82ca890-8cb0-42e1-9325-555f07de811d/kube-rbac-proxy/0.log"