Sep 08 16:50:56.414617 ip-10-0-133-106 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 16:50:56.834944 ip-10-0-133-106 kubenswrapper[2542]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:50:56.834944 ip-10-0-133-106 kubenswrapper[2542]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 16:50:56.834944 ip-10-0-133-106 kubenswrapper[2542]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:50:56.834944 ip-10-0-133-106 kubenswrapper[2542]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 16:50:56.834944 ip-10-0-133-106 kubenswrapper[2542]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:50:56.836162 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.836072    2542 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 16:50:56.840054 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840038    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.840054 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840054    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840058    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840061    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840064    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840066    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840069    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840074    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840078    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840081    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840084    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840087    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840090    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840093    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840095    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840098    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840100    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840103    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840106    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840109    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.840121 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840112    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840114    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840117    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840120    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840122    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840126    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840129    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840132    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840135    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840137    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840140    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840142    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840145    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840148    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840151    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840153    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840155    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840158    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840160    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840163    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.840589 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840165    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840168    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840172    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840176    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840179    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840181    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840185    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840187    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840190    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840192    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840195    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840197    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840200    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840203    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840206    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840209    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840211    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840214    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840216    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840219    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.841066 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840222    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840225    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840229    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840232    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840235    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840237    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840240    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840243    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840245    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840248    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840250    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840253    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840255    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840258    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840261    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840263    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840265    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840270    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840272    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840275    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.841573 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840277    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840280    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840282    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840285    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840288    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840291    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840295    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840297    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840300    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840302    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840305    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840307    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840310    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840312    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840315    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840318    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840334    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840338    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.840340    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.842078 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841025    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841031    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841033    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841036    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841039    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841041    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841044    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841048    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841051    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841053    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841075    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841080    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841084    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841087    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841090    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841093    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841096    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841100    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841105    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.842655 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841109    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841112    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841115    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841118    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841120    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841123    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841126    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841128    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841131    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841134    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841136    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841139    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841141    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841144    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841146    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841149    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841152    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841154    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841157    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841160    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.843149 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841163    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841166    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841168    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841171    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841174    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841177    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841179    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841182    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841184    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841187    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841190    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841193    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841195    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841198    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841200    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841203    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841205    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841208    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841210    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841212    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.843666 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841215    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841217    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841220    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841223    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841225    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841228    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841230    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841233    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841237    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841241    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841244    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841247    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841250    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841253    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841256    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841258    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841261    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841264    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841267    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841270    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.844161 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841273    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841276    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841279    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841281    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841283    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841286    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841289    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841291    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841293    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841296    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841298    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841301    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841303    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841306    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841308    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841310    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841313    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841315    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841318    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841336    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.844656 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841706    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841711    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841714    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841716    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841719    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841722    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841725    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841728    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841730    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841733    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841736    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841738    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841741    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841744    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841747    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841750    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841753    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841755    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841758    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841760    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.845168 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841763    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841766    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841769    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841773    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841776    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841778    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841781    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841783    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841786    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841789    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841791    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841794    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841797    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841800    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841802    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841805    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841807    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841810    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841813    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841815    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.845678 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841818    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841820    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841823    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841826    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841831    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841834    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841836    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841839    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841841    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841844    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841846    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841849    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841851    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841854    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841856    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841859    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841861    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841864    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841866    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841869    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.846200 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841871    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841876    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841879    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841882    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841885    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841888    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841890    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841893    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841895    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841898    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841901    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841903    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841906    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841908    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841911    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841914    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841920    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841922    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841925    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.846716 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841929    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841932    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841935    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841938    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841941    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841943    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841946    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841949    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841951    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841954    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841957    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841960    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841963    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841966    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841968    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841971    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841973    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841976    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841978    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.841981    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.847183 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842056    2542 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842093    2542 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842118    2542 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842130    2542 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842135    2542 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842139    2542 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842144    2542 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842149    2542 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842152    2542 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842155    2542 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842159    2542 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842162    2542 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842165    2542 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842168    2542 flags.go:64] FLAG: --cgroup-root=""
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842171    2542 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842175    2542 flags.go:64] FLAG: --client-ca-file=""
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842178    2542 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842181    2542 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842186    2542 flags.go:64] FLAG: --cluster-domain=""
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842189    2542 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842193    2542 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842196    2542 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842199    2542 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842203    2542 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 16:50:56.847725 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842206    2542 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842209    2542 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842212    2542 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842216    2542 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842219    2542 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842222    2542 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842225    2542 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842228    2542 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842232    2542 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842235    2542 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842238    2542 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842241    2542 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842244    2542 flags.go:64] FLAG: --enable-server="true"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842247    2542 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842252    2542 flags.go:64] FLAG: --event-burst="100"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842255    2542 flags.go:64] FLAG: --event-qps="50"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842258    2542 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842261    2542 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842264    2542 flags.go:64] FLAG: --eviction-hard=""
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842268    2542 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842271    2542 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842274    2542 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842276    2542 flags.go:64] FLAG: --eviction-soft=""
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842280    2542 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842283    2542 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 16:50:56.848295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842285    2542 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842289    2542 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842292    2542 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842295    2542 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842298    2542 flags.go:64] FLAG: --feature-gates=""
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842302    2542 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842305    2542 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842308    2542 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842311    2542 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842314    2542 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842317    2542 flags.go:64] FLAG: --help="false"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842336    2542 flags.go:64] FLAG: --hostname-override="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842340    2542 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842343    2542 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842346    2542 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842350    2542 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842353    2542 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842359    2542 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842362    2542 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842365    2542 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842368    2542 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842371    2542 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842375    2542 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842378    2542 flags.go:64] FLAG: --kube-reserved=""
Sep 08 16:50:56.848935 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842381    2542 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842384    2542 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842387    2542 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842390    2542 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842393    2542 flags.go:64] FLAG: --lock-file=""
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842396    2542 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842399    2542 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842402    2542 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842407    2542 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842410    2542 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842413    2542 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842416    2542 flags.go:64] FLAG: --logging-format="text"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842419    2542 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842423    2542 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842426    2542 flags.go:64] FLAG: --manifest-url=""
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842428    2542 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842433    2542 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842436    2542 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842440    2542 flags.go:64] FLAG: --max-pods="110"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842443    2542 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842446    2542 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842449    2542 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842452    2542 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842455    2542 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842458    2542 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 16:50:56.849733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842463    2542 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842469    2542 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842472    2542 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842475    2542 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842478    2542 flags.go:64] FLAG: --pod-cidr=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842481    2542 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842486    2542 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842489    2542 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842492    2542 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842495    2542 flags.go:64] FLAG: --port="10250"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842498    2542 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842501    2542 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0121c3255867cf9f9"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842504    2542 flags.go:64] FLAG: --qos-reserved=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842507    2542 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842510    2542 flags.go:64] FLAG: --register-node="true"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842513    2542 flags.go:64] FLAG: --register-with-taints=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842517    2542 flags.go:64] FLAG: --registry-burst="10"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842520    2542 flags.go:64] FLAG: --registry-qps="5"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842522    2542 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842525    2542 flags.go:64] FLAG: --reserved-memory=""
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842529    2542 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842533    2542 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842536    2542 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842539    2542 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842542    2542 flags.go:64] FLAG: --runonce="false"
Sep 08 16:50:56.850784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842545    2542 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842548    2542 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842551    2542 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842554    2542 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842559    2542 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842562    2542 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842565    2542 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842578    2542 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842582    2542 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842585    2542 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842588    2542 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842591    2542 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842594    2542 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842597    2542 flags.go:64] FLAG: --system-cgroups=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842600    2542 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842606    2542 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842609    2542 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842611    2542 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842616    2542 flags.go:64] FLAG: --tls-min-version=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842619    2542 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842622    2542 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842625    2542 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842628    2542 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842631    2542 flags.go:64] FLAG: --v="2"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842635    2542 flags.go:64] FLAG: --version="false"
Sep 08 16:50:56.851726 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842639    2542 flags.go:64] FLAG: --vmodule=""
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842643    2542 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.842646    2542 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842764    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842768    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842772    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842775    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842778    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842780    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842783    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842786    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842789    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842792    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842794    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842798    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842801    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842804    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842807    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842809    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.852402 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842812    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842815    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842818    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842820    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842823    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842825    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842828    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842830    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842833    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842837    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842840    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842843    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842845    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842848    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842851    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842853    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842856    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842858    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842861    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842863    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.852871 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842866    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842869    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842880    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842883    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842885    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842888    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842891    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842895    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842898    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842900    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842903    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842906    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842908    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842911    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842913    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842916    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842919    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842922    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842924    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.853383 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842927    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842931    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842934    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842937    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842940    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842943    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842945    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842948    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842950    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842953    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842955    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842958    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842961    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842963    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842966    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842968    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842971    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842973    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842976    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.853859 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842978    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842983    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842986    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842988    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842991    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842993    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842996    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.842998    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843000    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843004    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843008    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843010    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843013    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843016    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843018    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843021    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843024    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843027    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843029    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843032    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.854345 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843035    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843038    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843040    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843043    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.843045    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.843671    2542 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.852639    2542 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.852654    2542 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.852673    2542 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.852679    2542 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852736    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852740    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852744    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852747    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.854849 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852750    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852753    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852756    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852759    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852761    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852764    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852767    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852770    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852772    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852775    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852777    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852782    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852785    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852789    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852791    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852794    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852797    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852799    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852802    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852805    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.855204 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852808    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852810    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852813    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852816    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852825    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852829    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852832    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852836    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852839    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852841    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852844    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852847    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852849    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852852    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852855    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852857    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852860    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852863    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852865    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.855751 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852868    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852871    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852873    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852876    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852879    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852882    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852884    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852887    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852889    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852892    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852895    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852897    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852900    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852902    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852905    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852909    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852912    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852915    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852918    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.856206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852921    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852924    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852927    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852930    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852933    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852936    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852939    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852942    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852944    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852947    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852950    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852953    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852955    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852958    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852960    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852963    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852966    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852969    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852972    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852974    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.856693 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852977    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852979    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852982    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852984    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852987    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852989    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852991    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852994    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852997    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.852999    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853002    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853004    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853007    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853010    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853013    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853015    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.857180 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853018    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.853023    2542 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853137    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853142    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853145    2542 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853148    2542 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853152    2542 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853155    2542 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853158    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853161    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853164    2542 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853167    2542 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853170    2542 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853172    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853176    2542 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:50:56.857582 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853178    2542 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853181    2542 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853183    2542 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853186    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853189    2542 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853191    2542 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853194    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853196    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853199    2542 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853201    2542 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853204    2542 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853206    2542 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853209    2542 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853212    2542 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853214    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853217    2542 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853221    2542 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853224    2542 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853226    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:50:56.857966 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853229    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853231    2542 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853234    2542 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853237    2542 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853239    2542 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853242    2542 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853245    2542 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853247    2542 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853250    2542 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853252    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853255    2542 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853258    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853260    2542 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853264    2542 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853266    2542 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853270    2542 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853274    2542 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853277    2542 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853280    2542 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853283    2542 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:50:56.858451 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853287    2542 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853290    2542 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853294    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853296    2542 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853299    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853302    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853305    2542 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853309    2542 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853313    2542 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853317    2542 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853334    2542 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853338    2542 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853342    2542 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853345    2542 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853349    2542 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853352    2542 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853355    2542 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853357    2542 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853360    2542 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:50:56.858967 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853362    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853365    2542 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853368    2542 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853370    2542 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853373    2542 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853375    2542 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853378    2542 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853382    2542 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853384    2542 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853387    2542 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853390    2542 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853393    2542 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853396    2542 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853400    2542 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853403    2542 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853407    2542 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853410    2542 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853412    2542 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853415    2542 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:50:56.859541 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853418    2542 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853420    2542 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853423    2542 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853426    2542 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853429    2542 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853431    2542 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853434    2542 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853437    2542 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:56.853440    2542 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.853445    2542 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.853558    2542 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.858239    2542 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.859188    2542 server.go:1019] "Starting client certificate rotation"
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.859288    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:50:56.860019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.859337    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:50:56.861776 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.861762    2542 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:50:56.863043 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.863025    2542 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:50:56.871178 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.871162    2542 log.go:25] "Validated CRI v1 runtime API"
Sep 08 16:50:56.873941 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.873928    2542 log.go:25] "Validated CRI v1 image API"
Sep 08 16:50:56.874753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.874739    2542 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 16:50:56.877347 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.877312    2542 fs.go:135] Filesystem UUIDs: map[67461c42-d35c-40b3-878d-ca494583bb0a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c70257ea-a060-4414-951a-62228e1d1111:/dev/nvme0n1p4]
Sep 08 16:50:56.877422 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.877348    2542 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 16:50:56.883166 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.883033    2542 manager.go:216] Machine: {Timestamp:2026-09-08 16:50:56.881682089 +0000 UTC m=+0.348888273 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100411 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d12dfff40f131ea3d7dbbdb03460e SystemUUID:ec2d12df-ff40-f131-ea3d-7dbbdb03460e BootID:925bbece-8348-4148-9097-27f4373c8d58 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:3a:ce:11:59 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:3a:ce:11:59 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:86:67:46:ca:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 16:50:56.883166 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.883153    2542 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 16:50:56.883346 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.883277    2542 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 16:50:56.887317 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.887119    2542 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 16:50:56.887563 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.887337    2542 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 16:50:56.888079 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.888070    2542 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 16:50:56.888109 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.888083    2542 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 16:50:56.888109 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.888091    2542 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:50:56.888828 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.888818    2542 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:50:56.889542 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.889532    2542 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 16:50:56.890652 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.890635    2542 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:50:56.893098 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893086    2542 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:50:56.893212 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893202    2542 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 16:50:56.893246 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893232    2542 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 16:50:56.893246 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893241    2542 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 16:50:56.893305 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893251    2542 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 16:50:56.893305 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893262    2542 kubelet.go:398] "Adding apiserver pod source"
Sep 08 16:50:56.893305 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893270    2542 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 16:50:56.893662 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893650    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:50:56.893712 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.893667    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:50:56.895910 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.895895    2542 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 16:50:56.898141 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.898104    2542 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 16:50:56.898141 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.898134    2542 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 16:50:56.899867 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899853    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899873    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899881    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899886    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899895    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899901    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 16:50:56.899915 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899914    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 16:50:56.900071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899919    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 16:50:56.900071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899926    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 16:50:56.900071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899932    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 16:50:56.900071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899940    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 16:50:56.900071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.899946    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 16:50:56.900837 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.900828    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 16:50:56.900837 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.900838    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 16:50:56.903437 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.903418    2542 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-106.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 16:50:56.903524 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.903506    2542 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 16:50:56.903561 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.903502    2542 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 16:50:56.904671 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.904657    2542 server.go:1268] "Started kubelet"
Sep 08 16:50:56.904731 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.904709    2542 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 16:50:56.904815 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.904756    2542 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 16:50:56.904883 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.904869    2542 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 16:50:56.905705 ip-10-0-133-106 systemd[1]: Started Kubernetes Kubelet.
Sep 08 16:50:56.906894 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.906768    2542 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 16:50:56.907390 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.907378    2542 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 16:50:56.910713 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.910692    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 16:50:56.910864 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.910839    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmmwg"
Sep 08 16:50:56.911314 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.911295    2542 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 16:50:56.912115 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.911842    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:56.912115 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.911969    2542 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 16:50:56.912115 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.911995    2542 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 16:50:56.912115 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912070    2542 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 16:50:56.912350 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912136    2542 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 16:50:56.912350 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912143    2542 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 16:50:56.912780 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.912549    2542 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 16:50:56.912780 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912624    2542 factory.go:153] Registering CRI-O factory
Sep 08 16:50:56.912780 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912642    2542 factory.go:223] Registration of the crio container factory successfully
Sep 08 16:50:56.912780 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912750    2542 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 16:50:56.912780 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912782    2542 factory.go:55] Registering systemd factory
Sep 08 16:50:56.913039 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912791    2542 factory.go:223] Registration of the systemd container factory successfully
Sep 08 16:50:56.913039 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912807    2542 factory.go:103] Registering Raw factory
Sep 08 16:50:56.913039 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.912823    2542 manager.go:1196] Started watching for new ooms in manager
Sep 08 16:50:56.913174 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.911965    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-106.ec2.internal.18d366929e60c11c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-106.ec2.internal,UID:ip-10-0-133-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-106.ec2.internal,},FirstTimestamp:2026-09-08 16:50:56.904618268 +0000 UTC m=+0.371824455,LastTimestamp:2026-09-08 16:50:56.904618268 +0000 UTC m=+0.371824455,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-106.ec2.internal,}"
Sep 08 16:50:56.913269 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.913236    2542 manager.go:349] Starting recovery of all containers
Sep 08 16:50:56.916552 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.916519    2542 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 16:50:56.916666 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.916636    2542 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 16:50:56.918657 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.918632    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zmmwg"
Sep 08 16:50:56.924920 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.924904    2542 manager.go:354] Recovery completed
Sep 08 16:50:56.929758 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.929746    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:56.931496 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.931477    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:56.931564 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.931509    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:56.931564 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.931524    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:56.932050 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932038    2542 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 16:50:56.932050 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932045    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:56.932152 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932048    2542 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 16:50:56.932152 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932082    2542 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:50:56.932700 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932687    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:56.932760 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932715    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:56.932760 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:56.932732    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:56.933735 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:56.933665    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-106.ec2.internal.18d366929ffad144  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-106.ec2.internal,UID:ip-10-0-133-106.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-106.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-106.ec2.internal,},FirstTimestamp:2026-09-08 16:50:56.931492164 +0000 UTC m=+0.398698350,LastTimestamp:2026-09-08 16:50:56.931492164 +0000 UTC m=+0.398698350,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-106.ec2.internal,}"
Sep 08 16:50:57.012302 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.012268    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.079966 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.079936    2542 policy_none.go:49] "None policy: Start"
Sep 08 16:50:57.079966 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.079964    2542 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 16:50:57.080138 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.079979    2542 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 16:50:57.082127 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.082116    2542 policy_none.go:47] "Start"
Sep 08 16:50:57.108763 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.108720    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 16:50:57.109937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.109919    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 16:50:57.110033 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.109970    2542 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 16:50:57.110033 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.109994    2542 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 16:50:57.110117 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.110039    2542 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 08 16:50:57.111071 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111052    2542 manager.go:337] "Starting Device Plugin manager"
Sep 08 16:50:57.111192 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.111101    2542 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 16:50:57.111192 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111115    2542 server.go:87] "Starting device plugin registration server"
Sep 08 16:50:57.111575 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111561    2542 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 16:50:57.111679 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111574    2542 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 16:50:57.111737 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111731    2542 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 16:50:57.111824 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111805    2542 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 16:50:57.111824 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.111817    2542 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 16:50:57.112422 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.112403    2542 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:50:57.112610 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.112595    2542 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 16:50:57.112666 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.112633    2542 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.121310 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.121289    2542 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-106.ec2.internal\" not found" node="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.210508 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.210465    2542 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"]
Sep 08 16:50:57.210648 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.210585    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:57.211823 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.211799    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:57.212205 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212190    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:57.212279 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212224    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:57.212279 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212238    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:57.212723 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212705    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:57.212803 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212741    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:57.212803 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212756    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:57.212803 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212778    2542 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.212803 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212785    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.212942 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.212817    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.214656 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.214640    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:57.214753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.214739    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.214796 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.214769    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:57.215295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215281    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:57.215377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215297    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:57.215377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215307    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:57.215377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215317    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:57.215377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215318    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:57.215377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.215348    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:57.217517 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.217503    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.217589 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.217527    2542 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:50:57.218188 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.218174    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:50:57.218258 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.218200    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:50:57.218258 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.218215    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:50:57.220267 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.220254    2542 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.220359 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.220273    2542 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-106.ec2.internal\": node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.234093 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.234071    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.244054 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.244032    2542 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-106.ec2.internal\" not found" node="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.248288 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.248273    2542 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-106.ec2.internal\" not found" node="ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.313392 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.313368    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.313553 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.313398    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.313553 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.313418    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/984ceb6c332935eca9a7c33929be3059-config\") pod \"kube-apiserver-proxy-ip-10-0-133-106.ec2.internal\" (UID: \"984ceb6c332935eca9a7c33929be3059\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.313553 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.313492    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.313657 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.313555    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1e50258ecd3d73f1826786329d7359d6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal\" (UID: \"1e50258ecd3d73f1826786329d7359d6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.335149 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.335129    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.413807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.413742    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/984ceb6c332935eca9a7c33929be3059-config\") pod \"kube-apiserver-proxy-ip-10-0-133-106.ec2.internal\" (UID: \"984ceb6c332935eca9a7c33929be3059\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.413807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.413708    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/984ceb6c332935eca9a7c33929be3059-config\") pod \"kube-apiserver-proxy-ip-10-0-133-106.ec2.internal\" (UID: \"984ceb6c332935eca9a7c33929be3059\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.435841 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.435814    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.536159 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.536127    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.546002 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.545975    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.551649 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.551127    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.637096 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.637049    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.738351 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.738263    2542 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-106.ec2.internal\" not found"
Sep 08 16:50:57.816104 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.816080    2542 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:50:57.859650 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.859624    2542 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 16:50:57.860117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.859766    2542 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:50:57.860117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.859803    2542 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:50:57.895247 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.895224    2542 apiserver.go:52] "Watching apiserver"
Sep 08 16:50:57.910994 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.910960    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 16:50:57.912898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.912878    2542 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.915604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.915577    2542 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 16:50:57.915948 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.915927    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-s4fdf","kube-system/global-pull-secret-syncer-fp7hm","kube-system/konnectivity-agent-jj7qm","openshift-image-registry/node-ca-dxpmq","openshift-multus/multus-additional-cni-plugins-4gcjd","openshift-ovn-kubernetes/ovnkube-node-2j8w7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv","openshift-cluster-node-tuning-operator/tuned-cxr7m","openshift-dns/node-resolver-rnlsr","openshift-multus/multus-wr7bb","openshift-multus/network-metrics-daemon-2rxnz","openshift-network-diagnostics/network-check-target-6vkpg"]
Sep 08 16:50:57.916758 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.916731    2542 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:50:57.918557 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.918542    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:57.920767 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.920747    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:57.920870 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.920845    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:50:57.921159 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.921142    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.921243 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.921146    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.921243 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.921146    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wv25c\""
Sep 08 16:50:57.921357 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.921185    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 16:50:57.921633 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.921617    2542 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:50:57.923077 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.923061    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:57.924111 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.924086    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:45:57 +0000 UTC" deadline="2028-06-14 18:20:38.64146973 +0000 UTC"
Sep 08 16:50:57.924183 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.924110    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15481h29m40.717363637s"
Sep 08 16:50:57.925123 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.925107    2542 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:50:57.925199 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.925186    2542 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"
Sep 08 16:50:57.925390 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.925368    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:57.926086 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.926069    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5ngt\""
Sep 08 16:50:57.926086 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.926083    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 16:50:57.926202 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.926106    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 16:50:57.927842 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.927828    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4w25c\""
Sep 08 16:50:57.927842 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.927835    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.927923 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.927857    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 16:50:57.928036 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.928021    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:57.928101 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.928073    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.930387 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930371    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:57.930486 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930420    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.930689 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930667    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5k7wg\""
Sep 08 16:50:57.930813 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930707    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 16:50:57.930937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930921    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 16:50:57.930937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930934    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 16:50:57.931059 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.930942    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.932963 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.932943    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.933078 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.932992    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 16:50:57.933078 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.932995    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 16:50:57.933078 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.933047    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:57.933872 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.933858    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 16:50:57.933920 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.933859    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 16:50:57.934081 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.934066    2542 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:50:57.934137 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.934106    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4rwhf\""
Sep 08 16:50:57.934192 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.934166    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.935146 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.935130    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 16:50:57.935411 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.935381    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:57.935489 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.935468    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.935568 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.935490    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7kjj7\""
Sep 08 16:50:57.935783 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.935768    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.938774 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.938752    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.940018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.939032    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnjkd\""
Sep 08 16:50:57.940018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.939047    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.944707 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.944684    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:57.947281 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.947262    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4prjq\""
Sep 08 16:50:57.947359 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.947292    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 16:50:57.947475 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.947462    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 16:50:57.947636 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.947621    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:57.949873 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.949854    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 16:50:57.950088 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.950070    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:57.950191 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.950137    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:50:57.950421 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.950404    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pjnt\""
Sep 08 16:50:57.951028 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.951013    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmrxl"
Sep 08 16:50:57.952477 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.952460    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal"]
Sep 08 16:50:57.952527 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.952490    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal"]
Sep 08 16:50:57.952571 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.952559    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:50:57.952637 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:57.952621    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:50:57.958011 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:57.957993    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dmrxl"
Sep 08 16:50:58.013767 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.013743    2542 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 16:50:58.017365 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017340    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-systemd-units\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.017478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017370    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-systemd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.017478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017387    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.017478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017423    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-run\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.017478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017457    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpqk\" (UniqueName: \"kubernetes.io/projected/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-kube-api-access-thpqk\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.017647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017494    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-netns\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.017647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017533    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-etc-kubernetes\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.017647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017566    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-system-cni-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.017647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017584    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-config\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.017647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017629    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cnibin\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.017830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017670    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.017830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017719    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-modprobe-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.017830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017744    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-kube-api-access-kllkx\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.017830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017760    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2m5n\" (UniqueName: \"kubernetes.io/projected/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-kube-api-access-w2m5n\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.017830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017787    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-kubelet\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017825    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn77n\" (UniqueName: \"kubernetes.io/projected/9d93826e-f282-4eea-921c-f0c9d703c1d0-kube-api-access-kn77n\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017865    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017894    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-bin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017911    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-dbus\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017926    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-node-log\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017946    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017968    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da26b116-98f9-4269-afb0-d8c118b580a0-ovn-node-metrics-cert\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.017984    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-script-lib\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018013    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.018041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018041    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e04be129-fe5a-47c5-a4cc-969300cc8f85-agent-certs\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018082    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cnibin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018126    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-socket-dir-parent\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018156    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018183    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018207    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysconfig\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018235    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-conf\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018259    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-sys\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018285    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-etc-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018308    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018332    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8jt\" (UniqueName: \"kubernetes.io/projected/09a37bc3-ac82-4e82-a7c1-3412a315862d-kube-api-access-wq8jt\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018374    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp9j7\" (UniqueName: \"kubernetes.io/projected/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-kube-api-access-cp9j7\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018397    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-serviceca\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018422    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgzs\" (UniqueName: \"kubernetes.io/projected/bdd679a3-1c44-40bc-9c88-f97be23eec57-kube-api-access-msgzs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018470    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g4f\" (UniqueName: \"kubernetes.io/projected/da26b116-98f9-4269-afb0-d8c118b580a0-kube-api-access-b9g4f\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018497    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-var-lib-kubelet\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.018555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018521    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-host\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018552    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-tmp\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018576    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018601    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-var-lib-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018625    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-ovn\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018650    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-netd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018675    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-kubelet-config\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018703    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-k8s-cni-cncf-io\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018728    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-kubernetes\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018752    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-lib-modules\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018777    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-tuned\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018802    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cni-binary-copy\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018829    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018854    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-netns\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018877    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-env-overrides\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018902    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-device-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.019293 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018932    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-multus-certs\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018969    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.018996    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7339ccf-b993-493e-a451-b9f4551ce405-host-slash\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019015    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-slash\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019036    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-hosts-file\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019051    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-system-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019064    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-conf-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019086    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019122    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7339ccf-b993-493e-a451-b9f4551ce405-iptables-alerter-script\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019191    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-multus\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019235    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019265    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019289    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019312    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-tmp-dir\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019373    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-kubelet\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019400    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e04be129-fe5a-47c5-a4cc-969300cc8f85-konnectivity-ca\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.019898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019428    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-host\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019458    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-systemd\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019476    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-hostroot\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019491    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019523    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-daemon-config\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019551    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019573    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqpnm\" (UniqueName: \"kubernetes.io/projected/f7339ccf-b993-493e-a451-b9f4551ce405-kube-api-access-hqpnm\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019604    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-log-socket\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.020465 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.019620    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-bin\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.120414 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120368    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e04be129-fe5a-47c5-a4cc-969300cc8f85-agent-certs\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.120414 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120415    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cnibin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.120646 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120442    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-socket-dir-parent\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.120646 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120490    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cnibin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.120646 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120514    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-socket-dir-parent\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.120646 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120615    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120662    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120691    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysconfig\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120720    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-conf\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120749    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-sys\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120773    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-etc-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.120807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120790    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysconfig\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120853    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-etc-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.120873    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120883    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-sys\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120897    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-conf\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120926    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.120958    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:58.620931888 +0000 UTC m=+2.088138083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120990    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.120993    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8jt\" (UniqueName: \"kubernetes.io/projected/09a37bc3-ac82-4e82-a7c1-3412a315862d-kube-api-access-wq8jt\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121032    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp9j7\" (UniqueName: \"kubernetes.io/projected/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-kube-api-access-cp9j7\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.121063 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121061    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-serviceca\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121087    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgzs\" (UniqueName: \"kubernetes.io/projected/bdd679a3-1c44-40bc-9c88-f97be23eec57-kube-api-access-msgzs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121116    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g4f\" (UniqueName: \"kubernetes.io/projected/da26b116-98f9-4269-afb0-d8c118b580a0-kube-api-access-b9g4f\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121085    2542 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121144    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-var-lib-kubelet\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121169    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-host\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121191    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-tmp\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121203    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-binary-copy\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121217    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121251    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-var-lib-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121281    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-ovn\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121319    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-var-lib-kubelet\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121349    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-var-lib-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121424    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-netd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121503    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-host\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121538    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-kubelet-config\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121570    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-k8s-cni-cncf-io\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.121624 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121603    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-kubernetes\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121642    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-kubelet-config\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121659    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-lib-modules\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121687    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-tuned\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121689    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-netd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121711    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cni-binary-copy\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121738    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121746    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-k8s-cni-cncf-io\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121767    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-netns\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121780    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121793    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-env-overrides\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121831    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-device-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121821    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-serviceca\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121859    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-multus-certs\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121883    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121901    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-ovn\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121907    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7339ccf-b993-493e-a451-b9f4551ce405-host-slash\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.122539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121945    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f7339ccf-b993-493e-a451-b9f4551ce405-host-slash\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121963    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-slash\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121989    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-device-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.121991    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-hosts-file\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122027    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-multus-certs\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122036    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-system-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122076    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-conf-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122121    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122150    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7339ccf-b993-493e-a451-b9f4551ce405-iptables-alerter-script\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122195    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-multus\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122222    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122266    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122295    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122343    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-cni-binary-copy\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122360    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-tmp-dir\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122389    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-kubelet\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122432    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e04be129-fe5a-47c5-a4cc-969300cc8f85-konnectivity-ca\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.123420 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122447    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-env-overrides\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122459    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-host\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122493    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-systemd\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122515    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-hostroot\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122528    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-slash\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122536    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122605    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-hosts-file\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122652    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-netns\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122689    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-daemon-config\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122713    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-system-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122723    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122728    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-kubernetes\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122753    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqpnm\" (UniqueName: \"kubernetes.io/projected/f7339ccf-b993-493e-a451-b9f4551ce405-kube-api-access-hqpnm\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122784    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-log-socket\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122771    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-conf-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122827    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122828    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-lib-modules\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122862    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-cni-dir\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.124302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122877    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-registration-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.122913    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-sys-fs\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.122944    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.122994    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:58.622976305 +0000 UTC m=+2.090182493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123084    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123081    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-log-socket\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123141    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-systemd\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123143    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-hostroot\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123184    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-kubelet\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123211    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-multus\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123370    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123512    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-tmp-dir\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123549    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-bin\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123580    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-host\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123584    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-systemd-units\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123617    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-systemd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123626    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-systemd-units\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125117 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123646    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123667    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-systemd\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123691    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f7339ccf-b993-493e-a451-b9f4551ce405-iptables-alerter-script\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123704    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-cni-bin\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123729    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-run\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123743    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-socket-dir\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123760    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpqk\" (UniqueName: \"kubernetes.io/projected/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-kube-api-access-thpqk\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123787    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-netns\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123796    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-run\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123809    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e04be129-fe5a-47c5-a4cc-969300cc8f85-konnectivity-ca\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123812    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-etc-kubernetes\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123845    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-run-netns\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123871    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-system-cni-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123900    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-config\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123925    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cnibin\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123954    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123873    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-etc-kubernetes\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123981    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-cnibin\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.125757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123980    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-modprobe-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.123975    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-system-cni-dir\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124011    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-run-openvswitch\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124025    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-kube-api-access-kllkx\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124057    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-modprobe-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124057    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2m5n\" (UniqueName: \"kubernetes.io/projected/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-kube-api-access-w2m5n\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124136    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-kubelet\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124165    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn77n\" (UniqueName: \"kubernetes.io/projected/9d93826e-f282-4eea-921c-f0c9d703c1d0-kube-api-access-kn77n\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124192    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124209    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-kubelet\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124220    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-bin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124246    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-dbus\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124270    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-node-log\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124295    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124323    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da26b116-98f9-4269-afb0-d8c118b580a0-ovn-node-metrics-cert\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124356    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-host-var-lib-cni-bin\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124379    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-script-lib\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.126604 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124389    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-multus-daemon-config\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124406    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124446    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-node-log\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124487    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-sysctl-d\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124489    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-config\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124516    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d93826e-f282-4eea-921c-f0c9d703c1d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124500    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c813616-6cba-4cb3-8d91-22703afd28c0-dbus\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124562    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da26b116-98f9-4269-afb0-d8c118b580a0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.124995    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/da26b116-98f9-4269-afb0-d8c118b580a0-ovnkube-script-lib\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.125578    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-tmp\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.126120    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/09a37bc3-ac82-4e82-a7c1-3412a315862d-etc-tuned\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.126482    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e04be129-fe5a-47c5-a4cc-969300cc8f85-agent-certs\") pod \"konnectivity-agent-jj7qm\" (UID: \"e04be129-fe5a-47c5-a4cc-969300cc8f85\") " pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.127383 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.126883    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/da26b116-98f9-4269-afb0-d8c118b580a0-ovn-node-metrics-cert\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.132865 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.132669    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp9j7\" (UniqueName: \"kubernetes.io/projected/1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f-kube-api-access-cp9j7\") pod \"node-resolver-rnlsr\" (UID: \"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f\") " pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.133652 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.133462    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:50:58.133652 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.133488    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:50:58.133652 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.133502    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:58.133652 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.133570    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:58.633546006 +0000 UTC m=+2.100752196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:58.135995 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.135967    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8jt\" (UniqueName: \"kubernetes.io/projected/09a37bc3-ac82-4e82-a7c1-3412a315862d-kube-api-access-wq8jt\") pod \"tuned-cxr7m\" (UID: \"09a37bc3-ac82-4e82-a7c1-3412a315862d\") " pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.135995 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.135994    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqpnm\" (UniqueName: \"kubernetes.io/projected/f7339ccf-b993-493e-a451-b9f4551ce405-kube-api-access-hqpnm\") pod \"iptables-alerter-s4fdf\" (UID: \"f7339ccf-b993-493e-a451-b9f4551ce405\") " pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.136283 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.136236    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g4f\" (UniqueName: \"kubernetes.io/projected/da26b116-98f9-4269-afb0-d8c118b580a0-kube-api-access-b9g4f\") pod \"ovnkube-node-2j8w7\" (UID: \"da26b116-98f9-4269-afb0-d8c118b580a0\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.136460 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.136427    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2m5n\" (UniqueName: \"kubernetes.io/projected/4c9cf456-163c-4f9b-ae20-b7f5ce64c673-kube-api-access-w2m5n\") pod \"multus-additional-cni-plugins-4gcjd\" (UID: \"4c9cf456-163c-4f9b-ae20-b7f5ce64c673\") " pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.137687 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.137657    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpqk\" (UniqueName: \"kubernetes.io/projected/798f9bdd-3b9f-408d-9bf2-cf639dcd2de6-kube-api-access-thpqk\") pod \"node-ca-dxpmq\" (UID: \"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6\") " pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.137898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.137877    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgzs\" (UniqueName: \"kubernetes.io/projected/bdd679a3-1c44-40bc-9c88-f97be23eec57-kube-api-access-msgzs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.138060 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.138045    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn77n\" (UniqueName: \"kubernetes.io/projected/9d93826e-f282-4eea-921c-f0c9d703c1d0-kube-api-access-kn77n\") pod \"aws-ebs-csi-driver-node-jfcdv\" (UID: \"9d93826e-f282-4eea-921c-f0c9d703c1d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.138547 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.138530    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kllkx\" (UniqueName: \"kubernetes.io/projected/9cb35f70-fad2-4215-a1c0-0f4be01c68ec-kube-api-access-kllkx\") pod \"multus-wr7bb\" (UID: \"9cb35f70-fad2-4215-a1c0-0f4be01c68ec\") " pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.246638 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.246580    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4fdf"
Sep 08 16:50:58.250844 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.250808    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984ceb6c332935eca9a7c33929be3059.slice/crio-2df07e7dcf9cce4a86fc1bd9fa5cf607eb30234d80178c243fe84a15ffef0666 WatchSource:0}: Error finding container 2df07e7dcf9cce4a86fc1bd9fa5cf607eb30234d80178c243fe84a15ffef0666: Status 404 returned error can't find the container with id 2df07e7dcf9cce4a86fc1bd9fa5cf607eb30234d80178c243fe84a15ffef0666
Sep 08 16:50:58.256161 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.256145    2542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:50:58.256808 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.256789    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7339ccf_b993_493e_a451_b9f4551ce405.slice/crio-7dcb5465d9870a6de787d7c00eca8c09138ba41aa051e195e3af84ad6ebb616e WatchSource:0}: Error finding container 7dcb5465d9870a6de787d7c00eca8c09138ba41aa051e195e3af84ad6ebb616e: Status 404 returned error can't find the container with id 7dcb5465d9870a6de787d7c00eca8c09138ba41aa051e195e3af84ad6ebb616e
Sep 08 16:50:58.269019 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.268999    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:50:58.278910 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.278889    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04be129_fe5a_47c5_a4cc_969300cc8f85.slice/crio-a6b4393299d453c1fe0fbd944b86b1d41495c029c95576bc6893be21563f6f4f WatchSource:0}: Error finding container a6b4393299d453c1fe0fbd944b86b1d41495c029c95576bc6893be21563f6f4f: Status 404 returned error can't find the container with id a6b4393299d453c1fe0fbd944b86b1d41495c029c95576bc6893be21563f6f4f
Sep 08 16:50:58.292555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.292524    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxpmq"
Sep 08 16:50:58.301033 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.301011    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798f9bdd_3b9f_408d_9bf2_cf639dcd2de6.slice/crio-9cd511365ba2224b8a6db0460b9776da6c052cc16568a37b2ffe4629efa230e2 WatchSource:0}: Error finding container 9cd511365ba2224b8a6db0460b9776da6c052cc16568a37b2ffe4629efa230e2: Status 404 returned error can't find the container with id 9cd511365ba2224b8a6db0460b9776da6c052cc16568a37b2ffe4629efa230e2
Sep 08 16:50:58.306256 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.306237    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4gcjd"
Sep 08 16:50:58.314526 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.314505    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9cf456_163c_4f9b_ae20_b7f5ce64c673.slice/crio-c6b3d6e69cd2e0dc0d135a89a89f9c20333f88ecfa9734b8e481d572ea995060 WatchSource:0}: Error finding container c6b3d6e69cd2e0dc0d135a89a89f9c20333f88ecfa9734b8e481d572ea995060: Status 404 returned error can't find the container with id c6b3d6e69cd2e0dc0d135a89a89f9c20333f88ecfa9734b8e481d572ea995060
Sep 08 16:50:58.322953 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.322933    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:50:58.326900 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.326881    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv"
Sep 08 16:50:58.332950 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.332906    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda26b116_98f9_4269_afb0_d8c118b580a0.slice/crio-ac5cf9478a6821d56a23e5a18839a28090e4978e2ee55b782dc72929c591ed50 WatchSource:0}: Error finding container ac5cf9478a6821d56a23e5a18839a28090e4978e2ee55b782dc72929c591ed50: Status 404 returned error can't find the container with id ac5cf9478a6821d56a23e5a18839a28090e4978e2ee55b782dc72929c591ed50
Sep 08 16:50:58.338152 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.338130    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d93826e_f282_4eea_921c_f0c9d703c1d0.slice/crio-e405c61bd343687122fd59fac00a1a315f41863154ddf9946ce0f79e9a7760ba WatchSource:0}: Error finding container e405c61bd343687122fd59fac00a1a315f41863154ddf9946ce0f79e9a7760ba: Status 404 returned error can't find the container with id e405c61bd343687122fd59fac00a1a315f41863154ddf9946ce0f79e9a7760ba
Sep 08 16:50:58.340398 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.340380    2542 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:50:58.351753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.351732    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cxr7m"
Sep 08 16:50:58.361067 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.361046    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a37bc3_ac82_4e82_a7c1_3412a315862d.slice/crio-72076da9c6191921027a82b97e50c393c6868eb8ac80a42fe82923a55bd16146 WatchSource:0}: Error finding container 72076da9c6191921027a82b97e50c393c6868eb8ac80a42fe82923a55bd16146: Status 404 returned error can't find the container with id 72076da9c6191921027a82b97e50c393c6868eb8ac80a42fe82923a55bd16146
Sep 08 16:50:58.374414 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.374380    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rnlsr"
Sep 08 16:50:58.378488 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.378466    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wr7bb"
Sep 08 16:50:58.382958 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.382937    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca4fe31_13cd_4ee8_b7df_a200e5f77c3f.slice/crio-c0940d6f17571895423ac270ba900b6ee48f5e6807e5f6ba7697da9c95fc0b1f WatchSource:0}: Error finding container c0940d6f17571895423ac270ba900b6ee48f5e6807e5f6ba7697da9c95fc0b1f: Status 404 returned error can't find the container with id c0940d6f17571895423ac270ba900b6ee48f5e6807e5f6ba7697da9c95fc0b1f
Sep 08 16:50:58.387682 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.387662    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb35f70_fad2_4215_a1c0_0f4be01c68ec.slice/crio-e01f90613c8a5d3de60f1509fae17deff3149eb34da065da2e1fd5d9ea05aabf WatchSource:0}: Error finding container e01f90613c8a5d3de60f1509fae17deff3149eb34da065da2e1fd5d9ea05aabf: Status 404 returned error can't find the container with id e01f90613c8a5d3de60f1509fae17deff3149eb34da065da2e1fd5d9ea05aabf
Sep 08 16:50:58.513192 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:50:58.513166    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e50258ecd3d73f1826786329d7359d6.slice/crio-a97a2c66f1f69677f1fbfbf9c444687cad39a38c76da1cd611aeabba63ad004a WatchSource:0}: Error finding container a97a2c66f1f69677f1fbfbf9c444687cad39a38c76da1cd611aeabba63ad004a: Status 404 returned error can't find the container with id a97a2c66f1f69677f1fbfbf9c444687cad39a38c76da1cd611aeabba63ad004a
Sep 08 16:50:58.627898 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.627859    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:58.628099 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.627915    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:58.628099 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.628030    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:58.628099 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.628037    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:58.628099 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.628092    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:59.62807765 +0000 UTC m=+3.095283822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:58.628294 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.628108    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:59.628099757 +0000 UTC m=+3.095305931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:58.713842 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.713812    2542 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:50:58.728741 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.728708    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:50:58.728964 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.728934    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:50:58.728964 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.728959    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:50:58.729087 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.728973    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:58.729087 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:58.729042    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:50:59.729021137 +0000 UTC m=+3.196227337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:58.961530 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.961424    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:45:58 +0000 UTC" deadline="2028-02-26 13:51:37.027199629 +0000 UTC"
Sep 08 16:50:58.961530 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:58.961472    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12861h0m38.065732783s"
Sep 08 16:50:59.113066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.113028    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:59.113221 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.113175    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:50:59.119217 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.119112    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal" event={"ID":"1e50258ecd3d73f1826786329d7359d6","Type":"ContainerStarted","Data":"a97a2c66f1f69677f1fbfbf9c444687cad39a38c76da1cd611aeabba63ad004a"}
Sep 08 16:50:59.123453 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.123399    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wr7bb" event={"ID":"9cb35f70-fad2-4215-a1c0-0f4be01c68ec","Type":"ContainerStarted","Data":"e01f90613c8a5d3de60f1509fae17deff3149eb34da065da2e1fd5d9ea05aabf"}
Sep 08 16:50:59.129283 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.129117    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rnlsr" event={"ID":"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f","Type":"ContainerStarted","Data":"c0940d6f17571895423ac270ba900b6ee48f5e6807e5f6ba7697da9c95fc0b1f"}
Sep 08 16:50:59.130824 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.130621    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cxr7m" event={"ID":"09a37bc3-ac82-4e82-a7c1-3412a315862d","Type":"ContainerStarted","Data":"72076da9c6191921027a82b97e50c393c6868eb8ac80a42fe82923a55bd16146"}
Sep 08 16:50:59.132938 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.132911    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"ac5cf9478a6821d56a23e5a18839a28090e4978e2ee55b782dc72929c591ed50"}
Sep 08 16:50:59.136130 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.136063    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerStarted","Data":"c6b3d6e69cd2e0dc0d135a89a89f9c20333f88ecfa9734b8e481d572ea995060"}
Sep 08 16:50:59.138111 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.138087    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxpmq" event={"ID":"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6","Type":"ContainerStarted","Data":"9cd511365ba2224b8a6db0460b9776da6c052cc16568a37b2ffe4629efa230e2"}
Sep 08 16:50:59.140556 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.140532    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jj7qm" event={"ID":"e04be129-fe5a-47c5-a4cc-969300cc8f85","Type":"ContainerStarted","Data":"a6b4393299d453c1fe0fbd944b86b1d41495c029c95576bc6893be21563f6f4f"}
Sep 08 16:50:59.143578 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.143523    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv" event={"ID":"9d93826e-f282-4eea-921c-f0c9d703c1d0","Type":"ContainerStarted","Data":"e405c61bd343687122fd59fac00a1a315f41863154ddf9946ce0f79e9a7760ba"}
Sep 08 16:50:59.146623 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.146596    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4fdf" event={"ID":"f7339ccf-b993-493e-a451-b9f4551ce405","Type":"ContainerStarted","Data":"7dcb5465d9870a6de787d7c00eca8c09138ba41aa051e195e3af84ad6ebb616e"}
Sep 08 16:50:59.149866 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.149840    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal" event={"ID":"984ceb6c332935eca9a7c33929be3059","Type":"ContainerStarted","Data":"2df07e7dcf9cce4a86fc1bd9fa5cf607eb30234d80178c243fe84a15ffef0666"}
Sep 08 16:50:59.636409 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.636220    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:50:59.636409 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.636299    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:50:59.636671 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.636450    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:59.636671 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.636520    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:01.636498007 +0000 UTC m=+5.103704199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:50:59.636969 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.636950    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:59.637032 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.637008    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:01.636992157 +0000 UTC m=+5.104198334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:50:59.736681 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.736640    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:50:59.736879 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.736846    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:50:59.736879 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.736872    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:50:59.736997 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.736886    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:59.736997 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:50:59.736951    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:01.736930154 +0000 UTC m=+5.204136331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:50:59.962132 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.962004    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:45:58 +0000 UTC" deadline="2028-03-06 03:22:45.587437889 +0000 UTC"
Sep 08 16:50:59.962132 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:50:59.962080    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13066h31m45.625362818s"
Sep 08 16:51:00.110689 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:00.110653    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:00.110885 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:00.110830    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:00.111294 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:00.111272    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:00.111431 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:00.111390    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:00.928756 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:00.928478    2542 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:51:01.111652 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:01.111619    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:01.112150 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.111762    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:01.652432 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:01.651671    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:01.652432 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:01.651745    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:01.652432 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.651883    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:01.652432 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.651946    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:05.651926553 +0000 UTC m=+9.119132727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:01.652893 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.652857    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:01.652993 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.652931    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:05.652911828 +0000 UTC m=+9.120118015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:01.752765 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:01.752723    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:01.752962 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.752919    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:51:01.752962 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.752940    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:51:01.752962 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.752952    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:01.753125 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:01.753014    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:05.752995283 +0000 UTC m=+9.220201454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:02.111245 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:02.111207    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:02.111467 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:02.111361    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:02.111796 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:02.111774    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:02.112306 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:02.111891    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:03.113479 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:03.112945    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:03.113479 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:03.113086    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:04.111629 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:04.111078    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:04.111629 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:04.111234    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:04.111629 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:04.111458    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:04.111629 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:04.111594    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:05.110798 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:05.110757    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:05.111315 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.110900    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:05.692439 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:05.692396    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:05.692639 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:05.692464    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:05.692639 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.692598    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:05.692754 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.692609    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:05.692754 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.692681    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.692658813 +0000 UTC m=+17.159864998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:05.692754 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.692719    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.692698132 +0000 UTC m=+17.159904307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:05.793090 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:05.793052    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:05.793268 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.793219    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:51:05.793268 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.793244    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:51:05.793268 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.793256    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:05.793434 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:05.793343    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:13.793307572 +0000 UTC m=+17.260513761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:06.111138 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:06.111102    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:06.111622 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:06.111146    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:06.111622 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:06.111259    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:06.111622 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:06.111385    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:07.111256 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:07.111219    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:07.111728 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:07.111342    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:08.110573 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:08.110538    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:08.110746 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:08.110589    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:08.110746 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:08.110673    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:08.110865 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:08.110758    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:09.110269 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:09.110232    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:09.110887 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:09.110376    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:10.110410 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:10.110318    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:10.110820 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:10.110457    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:10.110820 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:10.110526    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:10.110820 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:10.110651    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:11.110400 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:11.110370    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:11.110564 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:11.110504    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:12.110555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:12.110521    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:12.111014 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:12.110521    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:12.111014 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:12.110631    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:12.111014 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:12.110713    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:13.110619 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:13.110576    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:13.111040 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.110717    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:13.749251 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:13.749221    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:13.749421 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:13.749262    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:13.749421 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.749394    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:13.749421 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.749405    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:13.749519 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.749441    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.749427183 +0000 UTC m=+33.216633354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:51:13.749519 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.749454    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret podName:8c813616-6cba-4cb3-8d91-22703afd28c0 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.749448205 +0000 UTC m=+33.216654377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret") pod "global-pull-secret-syncer-fp7hm" (UID: "8c813616-6cba-4cb3-8d91-22703afd28c0") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:51:13.850278 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:13.850247    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:13.850440 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.850386    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:51:13.850440 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.850404    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:51:13.850440 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.850412    2542 projected.go:196] Error preparing data for projected volume kube-api-access-pzp7r for pod openshift-network-diagnostics/network-check-target-6vkpg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:13.850539 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:13.850461    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r podName:c495f225-0625-46a8-b6b9-1901c3aa50a6 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.850445042 +0000 UTC m=+33.317651232 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pzp7r" (UniqueName: "kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r") pod "network-check-target-6vkpg" (UID: "c495f225-0625-46a8-b6b9-1901c3aa50a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:51:14.110896 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:14.110814    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:14.110896 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:14.110859    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:14.111458 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:14.110945    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:14.111458 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:14.111057    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:15.110784 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:15.110746    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:15.110944 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:15.110870    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:16.110239 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:16.110205    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:16.110239 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:16.110226    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:16.110477 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:16.110336    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:16.110477 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:16.110430    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:17.110984 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:17.110950    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:17.111514 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:17.111082    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:18.111108 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.110951    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:18.111478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.110974    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:18.111478 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:18.111209    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:18.111478 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:18.111242    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:18.192811 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.192773    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal" event={"ID":"984ceb6c332935eca9a7c33929be3059","Type":"ContainerStarted","Data":"0387903e4b2de2f5a4b2355667283a52f710ea7d16a177cc406d44ef826ad442"}
Sep 08 16:51:18.194369 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.194341    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cxr7m" event={"ID":"09a37bc3-ac82-4e82-a7c1-3412a315862d","Type":"ContainerStarted","Data":"3a11570a3e275695f4f5df0ef281cefc234746d25eb768d4a5301c12659f3a31"}
Sep 08 16:51:18.196311 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.196291    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"70e7965a26a206d10430acfeed16ffa598c4fa81344d38fa4a6f9b8d574d93ef"}
Sep 08 16:51:18.196421 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.196367    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"07fd2f656839de4a0cef6f331e2a213a54f2f580987f601b4f74365fa99f8e79"}
Sep 08 16:51:18.228693 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.228651    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-106.ec2.internal" podStartSLOduration=20.228638353 podStartE2EDuration="20.228638353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:18.210632058 +0000 UTC m=+21.677838252" watchObservedRunningTime="2026-09-08 16:51:18.228638353 +0000 UTC m=+21.695844545"
Sep 08 16:51:18.228826 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:18.228802    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cxr7m" podStartSLOduration=1.8420510509999999 podStartE2EDuration="21.228792539s" totalImagesPullingTime="19.386741488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:18.228418364 +0000 UTC m=+21.695624556" watchObservedRunningTime="2026-09-08 16:51:18.228792539 +0000 UTC m=+21.695998733"
Sep 08 16:51:19.110354 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.110292    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:19.110489 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:19.110445    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:19.201507 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.201481    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.201817    2542 generic.go:356] "Generic (PLEG): container finished" podID="da26b116-98f9-4269-afb0-d8c118b580a0" containerID="70e7965a26a206d10430acfeed16ffa598c4fa81344d38fa4a6f9b8d574d93ef" exitCode=1
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.201887    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerDied","Data":"70e7965a26a206d10430acfeed16ffa598c4fa81344d38fa4a6f9b8d574d93ef"}
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.201951    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"22caeb50e678eae72f9c48dbc9d0d9ef79673901b40c3389d50c9b25ed9b679c"}
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.201987    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"6694b76d080a093023190b4d2c05848498d5f2b2489d09301ec1276d27d02173"}
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.202014    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"f74a0e1cbc0671da0c390cb668d005336905d55f590e0001510e391b5de57ee4"}
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.202048    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"dd12cbc204f6528f562eae59a2c6f37ac209d52f004a93f48fff4fd0f89f9d7a"}
Sep 08 16:51:19.202206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.202074    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"633835f79df09ce82a1c2d24eff4b9f0aecf49b1db9d5732e3de517b2e996029"}
Sep 08 16:51:19.203334 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.203299    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="d1a55ecbc849b191dd80c9faa6c5bdd9a4fbf672d8d044a2450545f4d124468e" exitCode=0
Sep 08 16:51:19.203445 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.203358    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"d1a55ecbc849b191dd80c9faa6c5bdd9a4fbf672d8d044a2450545f4d124468e"}
Sep 08 16:51:19.204921 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.204900    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxpmq" event={"ID":"798f9bdd-3b9f-408d-9bf2-cf639dcd2de6","Type":"ContainerStarted","Data":"5643bf07fef33132788570770cab1924a08a399627a8863eaf0868f07c91d203"}
Sep 08 16:51:19.206340 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.206308    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jj7qm" event={"ID":"e04be129-fe5a-47c5-a4cc-969300cc8f85","Type":"ContainerStarted","Data":"76f370b1330ec71af95bd6633cde15189dcfcc6b2e725e1aea0219eaafab4ec3"}
Sep 08 16:51:19.207857 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.207838    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv" event={"ID":"9d93826e-f282-4eea-921c-f0c9d703c1d0","Type":"ContainerStarted","Data":"09e343258198235e5c55b84c5dd73ba3841a40f05e3e08ce3fcbe133e4f4f22b"}
Sep 08 16:51:19.209760 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.209741    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4fdf" event={"ID":"f7339ccf-b993-493e-a451-b9f4551ce405","Type":"ContainerStarted","Data":"35bffe8a4ff71e732a6456d008494041124b3b40d35de17f0f1c74a6e0eefdce"}
Sep 08 16:51:19.211158 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.211141    2542 generic.go:356] "Generic (PLEG): container finished" podID="1e50258ecd3d73f1826786329d7359d6" containerID="e33ec164478d1582cc5f0f18906706b24808246f1d5dec887f113c9c22938ee1" exitCode=0
Sep 08 16:51:19.211226 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.211197    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal" event={"ID":"1e50258ecd3d73f1826786329d7359d6","Type":"ContainerDied","Data":"e33ec164478d1582cc5f0f18906706b24808246f1d5dec887f113c9c22938ee1"}
Sep 08 16:51:19.212736 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.212712    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wr7bb" event={"ID":"9cb35f70-fad2-4215-a1c0-0f4be01c68ec","Type":"ContainerStarted","Data":"38234ed7d35d0c2e7cf1f943f5767e2ed1c674f2c0f96bcfd060f3c520837bb8"}
Sep 08 16:51:19.214197 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.214172    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rnlsr" event={"ID":"1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f","Type":"ContainerStarted","Data":"c854358fa4e804a37b2223d87f4a146b2e97f2446be3245c17e127d6541571e9"}
Sep 08 16:51:19.236981 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.236946    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dxpmq" podStartSLOduration=2.833019466 podStartE2EDuration="22.236933724s" totalImagesPullingTime="19.403914258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:19.236718955 +0000 UTC m=+22.703925149" watchObservedRunningTime="2026-09-08 16:51:19.236933724 +0000 UTC m=+22.704139958"
Sep 08 16:51:19.257745 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.257686    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jj7qm" podStartSLOduration=2.78892544 podStartE2EDuration="22.257650894s" totalImagesPullingTime="19.468725454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:19.257542885 +0000 UTC m=+22.724749081" watchObservedRunningTime="2026-09-08 16:51:19.257650894 +0000 UTC m=+22.724857088"
Sep 08 16:51:19.292214 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.292176    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rnlsr" podStartSLOduration=2.970139698 podStartE2EDuration="22.292160716s" totalImagesPullingTime="19.322021018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:19.274312659 +0000 UTC m=+22.741518852" watchObservedRunningTime="2026-09-08 16:51:19.292160716 +0000 UTC m=+22.759366908"
Sep 08 16:51:19.292356 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.292258    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wr7bb" podStartSLOduration=2.336559802 podStartE2EDuration="22.292248322s" totalImagesPullingTime="19.95568852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:19.29171773 +0000 UTC m=+22.758923925" watchObservedRunningTime="2026-09-08 16:51:19.292248322 +0000 UTC m=+22.759454516"
Sep 08 16:51:19.323037 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.322992    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s4fdf" podStartSLOduration=2.806174499 podStartE2EDuration="22.322975608s" totalImagesPullingTime="19.516801109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:19.322378592 +0000 UTC m=+22.789584786" watchObservedRunningTime="2026-09-08 16:51:19.322975608 +0000 UTC m=+22.790181801"
Sep 08 16:51:19.738016 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:19.737841    2542 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 16:51:20.111004 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.110926    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:20.111168 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:20.111024    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:20.111168 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.111085    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:20.111271 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:20.111199    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:20.123492 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.123368    2542 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:51:19.738010488Z","UUID":"98bbc2d3-8c0c-449c-88d0-96e639a05bc0","Handler":null,"Name":"","Endpoint":""}
Sep 08 16:51:20.124786 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.124772    2542 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 16:51:20.124853 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.124800    2542 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 16:51:20.218746 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.218712    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv" event={"ID":"9d93826e-f282-4eea-921c-f0c9d703c1d0","Type":"ContainerStarted","Data":"1ef7d5f73de26d412bd5c7018028c3539269400fbbc34b26edc456c75361fc2c"}
Sep 08 16:51:20.220690 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.220655    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal" event={"ID":"1e50258ecd3d73f1826786329d7359d6","Type":"ContainerStarted","Data":"7d1e8662603fe951a9ffa1d9088071ec495350866ef3123aa3cd3dc03da9a293"}
Sep 08 16:51:20.237526 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:20.237486    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-106.ec2.internal" podStartSLOduration=22.237472954 podStartE2EDuration="22.237472954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:20.237090787 +0000 UTC m=+23.704296980" watchObservedRunningTime="2026-09-08 16:51:20.237472954 +0000 UTC m=+23.704679147"
Sep 08 16:51:21.111235 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.111200    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:21.111455 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:21.111354    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:21.197850 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.197745    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:51:21.198462 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.198440    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:51:21.226354 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.226313    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 16:51:21.226821 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.226736    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"c4334bccf1273b562b472f1cbe306b1edf540e47317d61ef981f9e2aa50d47d7"}
Sep 08 16:51:21.227024 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.226999    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:21.227147 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.227067    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:21.227206 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.227187    2542 scope.go:117] "RemoveContainer" containerID="70e7965a26a206d10430acfeed16ffa598c4fa81344d38fa4a6f9b8d574d93ef"
Sep 08 16:51:21.229764 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.229741    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv" event={"ID":"9d93826e-f282-4eea-921c-f0c9d703c1d0","Type":"ContainerStarted","Data":"f35bbba4480e8e1b63fad76d10931e3a9bc5d7d4fcdd1d44f87852e6d8b08029"}
Sep 08 16:51:21.230519 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.230110    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:51:21.230519 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.230394    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jj7qm"
Sep 08 16:51:21.248929 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.248889    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:21.303007 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:21.302959    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jfcdv" podStartSLOduration=3.005181754 podStartE2EDuration="24.302941458s" totalImagesPullingTime="21.297759704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:21.302905298 +0000 UTC m=+24.770111492" watchObservedRunningTime="2026-09-08 16:51:21.302941458 +0000 UTC m=+24.770147648"
Sep 08 16:51:22.111263 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.111222    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:22.111263 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.111253    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:22.111602 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:22.111393    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:22.111602 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:22.111508    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:22.238390 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.238356    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 16:51:22.239478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.239364    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" event={"ID":"da26b116-98f9-4269-afb0-d8c118b580a0","Type":"ContainerStarted","Data":"cc56f7f4d561a32d6592cfc73f41141346fd32ded242c734da18a7cfd3e0384e"}
Sep 08 16:51:22.239758 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.239738    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:22.258199 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.258169    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:22.297007 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.296943    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7" podStartSLOduration=5.838352986 podStartE2EDuration="25.296922797s" totalImagesPullingTime="19.458569811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:22.270203817 +0000 UTC m=+25.737410006" watchObservedRunningTime="2026-09-08 16:51:22.296922797 +0000 UTC m=+25.764128994"
Sep 08 16:51:22.893853 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.893623    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fp7hm"]
Sep 08 16:51:22.894052 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.893960    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:22.894114 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:22.894080    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:22.897396 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.896984    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6vkpg"]
Sep 08 16:51:22.897396 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.897092    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:22.897396 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:22.897181    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:22.897716 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.897691    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2rxnz"]
Sep 08 16:51:22.897842 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:22.897796    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:22.897927 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:22.897902    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:24.110520 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:24.110479    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:24.111215 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:24.110590    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:25.111103 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:25.111074    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:25.111630 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:25.111173    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:25.111630 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:25.111277    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:25.111630 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:25.111418    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:25.247544 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:25.247511    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="715ac6d8d50b398f2d47ad8117e6818f830f6af219b0e0d64e5018fc1e668cbb" exitCode=0
Sep 08 16:51:25.247690 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:25.247551    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"715ac6d8d50b398f2d47ad8117e6818f830f6af219b0e0d64e5018fc1e668cbb"}
Sep 08 16:51:26.110386 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:26.110356    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:26.110934 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:26.110501    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fp7hm" podUID="8c813616-6cba-4cb3-8d91-22703afd28c0"
Sep 08 16:51:26.253116 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:26.253082    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="c6eb8e4641e1b7961fbb918b359eefd080fefa5c3ea1a6e50a6d99fabfb79b3c" exitCode=0
Sep 08 16:51:26.253518 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:26.253142    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"c6eb8e4641e1b7961fbb918b359eefd080fefa5c3ea1a6e50a6d99fabfb79b3c"}
Sep 08 16:51:27.111697 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.111512    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:27.111914 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.111584    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:27.111914 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.111768    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6vkpg" podUID="c495f225-0625-46a8-b6b9-1901c3aa50a6"
Sep 08 16:51:27.111914 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.111860    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rxnz" podUID="bdd679a3-1c44-40bc-9c88-f97be23eec57"
Sep 08 16:51:27.267301 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.267279    2542 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-106.ec2.internal" event="NodeReady"
Sep 08 16:51:27.267688 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.267454    2542 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 16:51:27.317229 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.317200    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-569c94b8d6-pchlf"]
Sep 08 16:51:27.329755 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.329735    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.332434 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.332417    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f9rmv\""
Sep 08 16:51:27.332512 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.332417    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 16:51:27.332765 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.332749    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 16:51:27.332823 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.332763    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 16:51:27.339797 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.339773    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 16:51:27.339981 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.339957    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6hvpz"]
Sep 08 16:51:27.342907 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.342886    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-569c94b8d6-pchlf"]
Sep 08 16:51:27.343005 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.342918    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jxlfm"]
Sep 08 16:51:27.343005 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.342989    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.345579 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.345551    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpm9l\""
Sep 08 16:51:27.345683 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.345591    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 16:51:27.345683 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.345600    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 16:51:27.345881 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.345863    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.351840 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.351812    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 16:51:27.351937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.351925    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qdttq\""
Sep 08 16:51:27.352248 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.351816    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 16:51:27.352248 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.351816    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 16:51:27.356555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.355723    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hvpz"]
Sep 08 16:51:27.356555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.356518    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxlfm"]
Sep 08 16:51:27.446722 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446679    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446729    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446768    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446797    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79d7\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446823    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-config-volume\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446841    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr7j\" (UniqueName: \"kubernetes.io/projected/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-kube-api-access-gmr7j\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.446902 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446877    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446923    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446942    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446959    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-tmp-dir\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.446992    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.447015    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-cert\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.447029    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q484k\" (UniqueName: \"kubernetes.io/projected/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-kube-api-access-q484k\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.447185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.447048    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548038 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.547999    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548075    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548101    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548124    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-tmp-dir\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548155    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548172    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-cert\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.548203 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548188    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q484k\" (UniqueName: \"kubernetes.io/projected/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-kube-api-access-q484k\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548214    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548265    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548290    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548310    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548310    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548359    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79d7\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548383    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-config-volume\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548409    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr7j\" (UniqueName: \"kubernetes.io/projected/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-kube-api-access-gmr7j\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.548443    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.548458    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.548506    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:28.048489491 +0000 UTC m=+31.515695683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:27.548534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.548530    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-tmp-dir\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.549059 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.548683    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:27.549059 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:27.548728    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:28.048713863 +0000 UTC m=+31.515920060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:27.549141 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.549119    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.549183 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.549152    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-config-volume\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.549432 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.549401    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.552141 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.552118    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.552245 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.552146    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.552245 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.552163    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-cert\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.560202 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.560169    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr7j\" (UniqueName: \"kubernetes.io/projected/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-kube-api-access-gmr7j\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:27.560289 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.560219    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79d7\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.560289 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.560280    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q484k\" (UniqueName: \"kubernetes.io/projected/fa00cca0-57a4-4301-9b8d-e9d09393ba5f-kube-api-access-q484k\") pod \"ingress-canary-jxlfm\" (UID: \"fa00cca0-57a4-4301-9b8d-e9d09393ba5f\") " pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.560439 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.560424    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:27.701526 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.701494    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jxlfm"
Sep 08 16:51:27.977341 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:27.977243    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jxlfm"]
Sep 08 16:51:27.981694 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:27.981667    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa00cca0_57a4_4301_9b8d_e9d09393ba5f.slice/crio-c26cde4677ef7590aea4e0422223dc2de57e3bd7039e8cb26ea7b9e75427f6ee WatchSource:0}: Error finding container c26cde4677ef7590aea4e0422223dc2de57e3bd7039e8cb26ea7b9e75427f6ee: Status 404 returned error can't find the container with id c26cde4677ef7590aea4e0422223dc2de57e3bd7039e8cb26ea7b9e75427f6ee
Sep 08 16:51:28.053104 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.053064    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:28.053104 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.053103    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:28.053290 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:28.053195    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:28.053290 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:28.053206    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:28.053290 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:28.053205    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:28.053290 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:28.053257    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.053243477 +0000 UTC m=+32.520449652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:28.053290 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:28.053271    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:29.053265761 +0000 UTC m=+32.520471932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:28.110185 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.110149    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:28.112566 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.112549    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:51:28.260859 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.260775    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="57efa7c2590ce09d98d21f7a68513a5a0a23996af5ef8534ea540014c54b13d0" exitCode=0
Sep 08 16:51:28.260985 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.260861    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"57efa7c2590ce09d98d21f7a68513a5a0a23996af5ef8534ea540014c54b13d0"}
Sep 08 16:51:28.261996 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:28.261970    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxlfm" event={"ID":"fa00cca0-57a4-4301-9b8d-e9d09393ba5f","Type":"ContainerStarted","Data":"c26cde4677ef7590aea4e0422223dc2de57e3bd7039e8cb26ea7b9e75427f6ee"}
Sep 08 16:51:29.062129 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.062086    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:29.062129 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.062133    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:29.062600 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.062254    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:29.062600 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.062289    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:29.062600 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.062304    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:29.062600 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.062342    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:31.06230888 +0000 UTC m=+34.529515055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:29.062600 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.062361    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:31.062349517 +0000 UTC m=+34.529555687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:29.110649 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.110614    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:29.110812 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.110620    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:29.113161 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.113134    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:51:29.114596 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.114261    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:51:29.114596 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.114284    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:51:29.114596 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.114299    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\""
Sep 08 16:51:29.114596 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.114397    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8582r\""
Sep 08 16:51:29.767971 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.767939    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:51:29.768152 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.768028    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:29.768152 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.768114    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 16:51:29.768381 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:29.768183    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs podName:bdd679a3-1c44-40bc-9c88-f97be23eec57 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:01.76816748 +0000 UTC m=+65.235373676 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs") pod "network-metrics-daemon-2rxnz" (UID: "bdd679a3-1c44-40bc-9c88-f97be23eec57") : secret "metrics-daemon-secret" not found
Sep 08 16:51:29.770796 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.770768    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c813616-6cba-4cb3-8d91-22703afd28c0-original-pull-secret\") pod \"global-pull-secret-syncer-fp7hm\" (UID: \"8c813616-6cba-4cb3-8d91-22703afd28c0\") " pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:29.869201 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.869157    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:29.872107 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.872082    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzp7r\" (UniqueName: \"kubernetes.io/projected/c495f225-0625-46a8-b6b9-1901c3aa50a6-kube-api-access-pzp7r\") pod \"network-check-target-6vkpg\" (UID: \"c495f225-0625-46a8-b6b9-1901c3aa50a6\") " pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:29.919069 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:29.919027    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fp7hm"
Sep 08 16:51:30.023004 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:30.022919    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:30.649108 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:30.649028    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fp7hm"]
Sep 08 16:51:30.652691 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:30.652653    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c813616_6cba_4cb3_8d91_22703afd28c0.slice/crio-d5a73911934172d718a2f74d7a849f38023b53a9049841596f21c31990519ccf WatchSource:0}: Error finding container d5a73911934172d718a2f74d7a849f38023b53a9049841596f21c31990519ccf: Status 404 returned error can't find the container with id d5a73911934172d718a2f74d7a849f38023b53a9049841596f21c31990519ccf
Sep 08 16:51:30.656379 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:30.656309    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6vkpg"]
Sep 08 16:51:30.659399 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:30.659376    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc495f225_0625_46a8_b6b9_1901c3aa50a6.slice/crio-38dbf09754866d9b6c98858a1989f905ffc050721267bec241d5e6ce39fe7d19 WatchSource:0}: Error finding container 38dbf09754866d9b6c98858a1989f905ffc050721267bec241d5e6ce39fe7d19: Status 404 returned error can't find the container with id 38dbf09754866d9b6c98858a1989f905ffc050721267bec241d5e6ce39fe7d19
Sep 08 16:51:31.079157 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.079122    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:31.079297 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.079195    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:31.079297 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.079278    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:31.079297 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.079284    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:31.079426 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.079302    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:31.079426 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.079343    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:35.079316078 +0000 UTC m=+38.546522249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:31.079426 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.079361    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:35.079348431 +0000 UTC m=+38.546554606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:31.264577 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.263549    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cwjkc"]
Sep 08 16:51:31.269601 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.268997    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.272571 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.272545    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cvbpd\""
Sep 08 16:51:31.272571 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.272545    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 08 16:51:31.272744 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.272549    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 08 16:51:31.282638 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.281396    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jxlfm" event={"ID":"fa00cca0-57a4-4301-9b8d-e9d09393ba5f","Type":"ContainerStarted","Data":"c70aa02071cc16311119f2322191ed860584022163c6b53487707bd505b0ddbf"}
Sep 08 16:51:31.283515 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.282976    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cwjkc"]
Sep 08 16:51:31.285341 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.285073    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fp7hm" event={"ID":"8c813616-6cba-4cb3-8d91-22703afd28c0","Type":"ContainerStarted","Data":"d5a73911934172d718a2f74d7a849f38023b53a9049841596f21c31990519ccf"}
Sep 08 16:51:31.288142 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.288088    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6vkpg" event={"ID":"c495f225-0625-46a8-b6b9-1901c3aa50a6","Type":"ContainerStarted","Data":"38dbf09754866d9b6c98858a1989f905ffc050721267bec241d5e6ce39fe7d19"}
Sep 08 16:51:31.312478 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.312389    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jxlfm" podStartSLOduration=1.900250151 podStartE2EDuration="4.312369539s" totalImagesPullingTime="2.412119388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:31.311952825 +0000 UTC m=+34.779159030" watchObservedRunningTime="2026-09-08 16:51:31.312369539 +0000 UTC m=+34.779575733"
Sep 08 16:51:31.384199 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.384143    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-nginx-conf\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.384391 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.384237    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.485010 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.484974    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-nginx-conf\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.485184 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.485036    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.485258 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.485198    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:31.485308 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.485264    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:31.985244854 +0000 UTC m=+35.452451052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:31.491007 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.490950    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-nginx-conf\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.989550 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:31.989506    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:31.990020 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.989683    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:31.990020 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:31.989760    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:32.989739268 +0000 UTC m=+36.456945444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:32.726873 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.726837    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"]
Sep 08 16:51:32.745969 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.745935    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"]
Sep 08 16:51:32.746138 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.746056    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"
Sep 08 16:51:32.748903 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.748874    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 16:51:32.749381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.749361    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 16:51:32.750120 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.750098    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fgvsr\""
Sep 08 16:51:32.892085 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.892053    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k99cv"]
Sep 08 16:51:32.898734 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.898691    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2bt\" (UniqueName: \"kubernetes.io/projected/3e8ecea5-0f73-48cc-b24c-c297127a12f1-kube-api-access-nx2bt\") pod \"migrator-fdbb8fdff-7lkqs\" (UID: \"3e8ecea5-0f73-48cc-b24c-c297127a12f1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"
Sep 08 16:51:32.910139 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.910106    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k99cv"]
Sep 08 16:51:32.910270 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.910253    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:32.912765 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.912741    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 16:51:32.912930 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.912911    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 16:51:32.913674 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.913242    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 16:51:32.913674 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.913418    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxh7j\""
Sep 08 16:51:32.913674 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.913466    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 16:51:32.999700 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999603    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2bt\" (UniqueName: \"kubernetes.io/projected/3e8ecea5-0f73-48cc-b24c-c297127a12f1-kube-api-access-nx2bt\") pod \"migrator-fdbb8fdff-7lkqs\" (UID: \"3e8ecea5-0f73-48cc-b24c-c297127a12f1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"
Sep 08 16:51:32.999700 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999680    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999716    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999746    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c70a95-9295-4368-a484-1c6dacf4d6e5-data-volume\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999788    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:32.999850    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36c70a95-9295-4368-a484-1c6dacf4d6e5-crio-socket\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:32.999963    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.000053    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:35.000030192 +0000 UTC m=+38.467236408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:33.000195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.000153    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g566p\" (UniqueName: \"kubernetes.io/projected/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-api-access-g566p\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.011445 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.011414    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2bt\" (UniqueName: \"kubernetes.io/projected/3e8ecea5-0f73-48cc-b24c-c297127a12f1-kube-api-access-nx2bt\") pod \"migrator-fdbb8fdff-7lkqs\" (UID: \"3e8ecea5-0f73-48cc-b24c-c297127a12f1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"
Sep 08 16:51:33.057586 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.057542    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"
Sep 08 16:51:33.101405 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101372    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g566p\" (UniqueName: \"kubernetes.io/projected/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-api-access-g566p\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101440    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101477    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c70a95-9295-4368-a484-1c6dacf4d6e5-data-volume\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101539 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101521    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101700 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.101638    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:51:33.101700 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.101660    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:33.101785 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.101709    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.60168973 +0000 UTC m=+37.068895920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : configmap "kube-rbac-proxy" not found
Sep 08 16:51:33.101785 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.101729    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:33.601718642 +0000 UTC m=+37.068924813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:33.101785 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101749    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36c70a95-9295-4368-a484-1c6dacf4d6e5-crio-socket\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101892    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c70a95-9295-4368-a484-1c6dacf4d6e5-data-volume\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.101937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.101923    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/36c70a95-9295-4368-a484-1c6dacf4d6e5-crio-socket\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.114537 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.114506    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g566p\" (UniqueName: \"kubernetes.io/projected/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-api-access-g566p\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.606251 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.606208    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.606434 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:33.606279    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:33.606434 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.606352    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:51:33.606434 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.606434    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:34.606413762 +0000 UTC m=+38.073619935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : configmap "kube-rbac-proxy" not found
Sep 08 16:51:33.606619 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.606438    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:33.606619 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:33.606499    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:34.606481758 +0000 UTC m=+38.073687929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:34.615895 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:34.615859    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:34.616317 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:34.615982    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:34.616317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:34.616015    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:34.616317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:34.616056    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:51:34.616317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:34.616081    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:36.616063684 +0000 UTC m=+40.083269865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:34.616317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:34.616097    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:36.616089496 +0000 UTC m=+40.083295668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : configmap "kube-rbac-proxy" not found
Sep 08 16:51:35.007577 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.007545    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wl245"]
Sep 08 16:51:35.018524 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.018487    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:35.018666 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.018641    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:35.018740 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.018717    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:39.018697347 +0000 UTC m=+42.485903541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:35.022430 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.022399    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wl245"]
Sep 08 16:51:35.022549 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.022508    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.025575 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.025546    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 16:51:35.025679 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.025579    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6xgwm\""
Sep 08 16:51:35.025679 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.025671    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 16:51:35.025802 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.025732    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 16:51:35.026230 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.026212    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 16:51:35.119165 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.119130    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-key\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.119364 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.119210    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-cabundle\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.119364 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.119268    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:35.119364 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.119296    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:35.119364 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.119342    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v8q\" (UniqueName: \"kubernetes.io/projected/884d48b6-b59d-48ff-90df-3171eb3fb85b-kube-api-access-s9v8q\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.119578 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.119416    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:35.119578 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.119457    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:35.119578 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.119463    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:43.119450202 +0000 UTC m=+46.586656376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:35.119578 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.119467    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:35.119578 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:35.119519    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:43.119502669 +0000 UTC m=+46.586708840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:35.219892 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.219856    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-key\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.220080 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.219908    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-cabundle\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.220080 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.219998    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v8q\" (UniqueName: \"kubernetes.io/projected/884d48b6-b59d-48ff-90df-3171eb3fb85b-kube-api-access-s9v8q\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.220773 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.220741    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-cabundle\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.222460 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.222440    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/884d48b6-b59d-48ff-90df-3171eb3fb85b-signing-key\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.229196 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.229164    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v8q\" (UniqueName: \"kubernetes.io/projected/884d48b6-b59d-48ff-90df-3171eb3fb85b-kube-api-access-s9v8q\") pod \"service-ca-75d99c484-wl245\" (UID: \"884d48b6-b59d-48ff-90df-3171eb3fb85b\") " pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.333708 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.333630    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-wl245"
Sep 08 16:51:35.425897 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:35.425869    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rnlsr_1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f/dns-node-resolver/0.log"
Sep 08 16:51:36.224873 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:36.224839    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxpmq_798f9bdd-3b9f-408d-9bf2-cf639dcd2de6/node-ca/0.log"
Sep 08 16:51:36.633125 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:36.633039    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:36.633125 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:36.633101    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:36.633317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:36.633176    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:51:36.633317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:36.633200    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:36.633317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:36.633242    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:40.633227583 +0000 UTC m=+44.100433773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : configmap "kube-rbac-proxy" not found
Sep 08 16:51:36.633317 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:36.633262    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:40.633248398 +0000 UTC m=+44.100454569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:36.826511 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:36.826485    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jxlfm_fa00cca0-57a4-4301-9b8d-e9d09393ba5f/serve-healthcheck-canary/0.log"
Sep 08 16:51:37.305286 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.305250    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fp7hm" event={"ID":"8c813616-6cba-4cb3-8d91-22703afd28c0","Type":"ContainerStarted","Data":"28e30cc6db7befbbe4c7745cdb61f0a6c5f82cd81b86b2204f5b507e136d88dc"}
Sep 08 16:51:37.306962 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.306936    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6vkpg" event={"ID":"c495f225-0625-46a8-b6b9-1901c3aa50a6","Type":"ContainerStarted","Data":"a22c3abb285f29e0ca4c874cf9039bac09a4addd104ec6e61462fee67911ebbd"}
Sep 08 16:51:37.307081 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.307020    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:51:37.309797 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.309741    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerStarted","Data":"bafeebdf5f82011d4c989496e0b9489f37f0f0ced9fddd92deaf7cb2205f3c8d"}
Sep 08 16:51:37.312500 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.311318    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-wl245"]
Sep 08 16:51:37.315389 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.315368    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs"]
Sep 08 16:51:37.322700 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.322657    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fp7hm" podStartSLOduration=33.939378 podStartE2EDuration="40.322643477s" totalImagesPullingTime="6.383265477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:37.32212704 +0000 UTC m=+40.789333237" watchObservedRunningTime="2026-09-08 16:51:37.322643477 +0000 UTC m=+40.789849671"
Sep 08 16:51:37.340206 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:37.340185    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884d48b6_b59d_48ff_90df_3171eb3fb85b.slice/crio-dba798331326f3e41bb2cad5c9f408d1f26f3119b78c4b017936f545cb22d12f WatchSource:0}: Error finding container dba798331326f3e41bb2cad5c9f408d1f26f3119b78c4b017936f545cb22d12f: Status 404 returned error can't find the container with id dba798331326f3e41bb2cad5c9f408d1f26f3119b78c4b017936f545cb22d12f
Sep 08 16:51:37.345130 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:37.345104    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8ecea5_0f73_48cc_b24c_c297127a12f1.slice/crio-125548034a01a52edaf087870f61f79c3dbd298050b4637e9837ceb224530b04 WatchSource:0}: Error finding container 125548034a01a52edaf087870f61f79c3dbd298050b4637e9837ceb224530b04: Status 404 returned error can't find the container with id 125548034a01a52edaf087870f61f79c3dbd298050b4637e9837ceb224530b04
Sep 08 16:51:37.361905 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:37.361862    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6vkpg" podStartSLOduration=33.975939294 podStartE2EDuration="40.361848648s" totalImagesPullingTime="6.385909354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:37.361537437 +0000 UTC m=+40.828743629" watchObservedRunningTime="2026-09-08 16:51:37.361848648 +0000 UTC m=+40.829054832"
Sep 08 16:51:38.316012 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:38.315920    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="bafeebdf5f82011d4c989496e0b9489f37f0f0ced9fddd92deaf7cb2205f3c8d" exitCode=0
Sep 08 16:51:38.316504 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:38.316002    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"bafeebdf5f82011d4c989496e0b9489f37f0f0ced9fddd92deaf7cb2205f3c8d"}
Sep 08 16:51:38.317404 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:38.317352    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-wl245" event={"ID":"884d48b6-b59d-48ff-90df-3171eb3fb85b","Type":"ContainerStarted","Data":"dba798331326f3e41bb2cad5c9f408d1f26f3119b78c4b017936f545cb22d12f"}
Sep 08 16:51:38.318750 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:38.318627    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs" event={"ID":"3e8ecea5-0f73-48cc-b24c-c297127a12f1","Type":"ContainerStarted","Data":"125548034a01a52edaf087870f61f79c3dbd298050b4637e9837ceb224530b04"}
Sep 08 16:51:39.054173 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:39.054137    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:39.054307 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:39.054264    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:39.054364 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:39.054340    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:51:47.054305155 +0000 UTC m=+50.521511350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:39.325345 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:39.325245    2542 generic.go:356] "Generic (PLEG): container finished" podID="4c9cf456-163c-4f9b-ae20-b7f5ce64c673" containerID="91180cd82a7c7806da1ef806bf0177b971fe8dd6460a637bc16289c6fdc23413" exitCode=0
Sep 08 16:51:39.325345 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:39.325293    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerDied","Data":"91180cd82a7c7806da1ef806bf0177b971fe8dd6460a637bc16289c6fdc23413"}
Sep 08 16:51:40.330801 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.330710    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs" event={"ID":"3e8ecea5-0f73-48cc-b24c-c297127a12f1","Type":"ContainerStarted","Data":"958e65691fa2414cfff12f6abbde0331e56f5a5dd31f46b68236f0caf09c76b3"}
Sep 08 16:51:40.331225 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.330805    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs" event={"ID":"3e8ecea5-0f73-48cc-b24c-c297127a12f1","Type":"ContainerStarted","Data":"53eea3c843cb2e45308fdf5a0fc60a18af575d6bf0f1fb05a55f059e285f6fdf"}
Sep 08 16:51:40.334396 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.334368    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" event={"ID":"4c9cf456-163c-4f9b-ae20-b7f5ce64c673","Type":"ContainerStarted","Data":"d1655089b8b64578ae05f79cbc9bb478457479c2bec731e21c802baa935d59e2"}
Sep 08 16:51:40.335799 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.335771    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-wl245" event={"ID":"884d48b6-b59d-48ff-90df-3171eb3fb85b","Type":"ContainerStarted","Data":"a3c74e745b487053f4150d3c464faeb3402fcb95594d5833b84e3ede466d1351"}
Sep 08 16:51:40.348012 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.347969    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-7lkqs" podStartSLOduration=6.065706436 podStartE2EDuration="8.34795507s" totalImagesPullingTime="2.282248634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:40.346165584 +0000 UTC m=+43.813371776" watchObservedRunningTime="2026-09-08 16:51:40.34795507 +0000 UTC m=+43.815161320"
Sep 08 16:51:40.366570 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.366518    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4gcjd" podStartSLOduration=8.019819079 podStartE2EDuration="43.366499012s" totalImagesPullingTime="35.346679933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:40.364936861 +0000 UTC m=+43.832143055" watchObservedRunningTime="2026-09-08 16:51:40.366499012 +0000 UTC m=+43.833705206"
Sep 08 16:51:40.380156 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.380114    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-wl245" podStartSLOduration=3.748523629 podStartE2EDuration="6.380099181s" totalImagesPullingTime="2.631575552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:40.379577127 +0000 UTC m=+43.846783311" watchObservedRunningTime="2026-09-08 16:51:40.380099181 +0000 UTC m=+43.847305375"
Sep 08 16:51:40.669834 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.669747    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:40.669834 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:40.669798    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:40.670008 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:40.669894    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:51:40.670008 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:40.669928    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:40.670008 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:40.669981    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:48.669958205 +0000 UTC m=+52.137164380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : configmap "kube-rbac-proxy" not found
Sep 08 16:51:40.670008 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:40.670001    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:48.6699914 +0000 UTC m=+52.137197576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:43.190929 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:43.190892    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:43.190929 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:43.190934    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:43.191348 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:43.191038    2542 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:51:43.191348 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:43.191050    2542 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-569c94b8d6-pchlf: secret "image-registry-tls" not found
Sep 08 16:51:43.191348 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:43.191056    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:51:43.191348 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:43.191098    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls podName:24ec4a68-9201-4c2f-927d-5901d2aa8a85 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:59.191085219 +0000 UTC m=+62.658291390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls") pod "image-registry-569c94b8d6-pchlf" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85") : secret "image-registry-tls" not found
Sep 08 16:51:43.191348 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:43.191111    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls podName:30ac2086-1fcf-4ec7-9ed2-8202a55eb804 nodeName:}" failed. No retries permitted until 2026-09-08 16:51:59.19110519 +0000 UTC m=+62.658311362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls") pod "dns-default-6hvpz" (UID: "30ac2086-1fcf-4ec7-9ed2-8202a55eb804") : secret "dns-default-metrics-tls" not found
Sep 08 16:51:47.123582 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:47.123544    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:51:47.124055 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:47.123680    2542 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 08 16:51:47.124055 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:47.123763    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert podName:6478c3cb-d09f-4aa6-bad7-5c87a320b03e nodeName:}" failed. No retries permitted until 2026-09-08 16:52:03.123730915 +0000 UTC m=+66.590937106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cwjkc" (UID: "6478c3cb-d09f-4aa6-bad7-5c87a320b03e") : secret "networking-console-plugin-cert" not found
Sep 08 16:51:48.736676 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:48.736620    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:48.737068 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:48.736764    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:48.737068 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:48.736767    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:48.737068 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:48.736873    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls podName:36c70a95-9295-4368-a484-1c6dacf4d6e5 nodeName:}" failed. No retries permitted until 2026-09-08 16:52:04.736853589 +0000 UTC m=+68.204059762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k99cv" (UID: "36c70a95-9295-4368-a484-1c6dacf4d6e5") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:51:48.737221 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:48.737199    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/36c70a95-9295-4368-a484-1c6dacf4d6e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:51:54.254794 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:54.254769    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j8w7"
Sep 08 16:51:57.811124 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.811083    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"]
Sep 08 16:51:57.815404 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.815386    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"]
Sep 08 16:51:57.815523 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.815505    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:57.818282 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818264    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 16:51:57.818519 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818491    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"]
Sep 08 16:51:57.818519 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818509    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 16:51:57.818670 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818539    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 16:51:57.818670 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818550    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4khvm\""
Sep 08 16:51:57.818670 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818558    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 16:51:57.818832 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.818687    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.821746 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.821730    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 16:51:57.821842 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.821803    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 16:51:57.821984 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.821964    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:57.822055 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.821990    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 16:51:57.822055 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.822020    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 16:51:57.824679 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.824660    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 16:51:57.824967 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.824935    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"]
Sep 08 16:51:57.826176 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.826150    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"]
Sep 08 16:51:57.832657 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.832634    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"]
Sep 08 16:51:57.904121 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904085    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-569c94b8d6-pchlf"]
Sep 08 16:51:57.904280 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904240    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e66ff-e961-4503-9752-b506feeeb46c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904280 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:51:57.904264    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-569c94b8d6-pchlf" podUID="24ec4a68-9201-4c2f-927d-5901d2aa8a85"
Sep 08 16:51:57.904280 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904271    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjd4\" (UniqueName: \"kubernetes.io/projected/979e66ff-e961-4503-9752-b506feeeb46c-kube-api-access-ffjd4\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904473 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904297    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904473 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904412    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kgq\" (UniqueName: \"kubernetes.io/projected/5cdb98e3-4600-4db4-9ab9-54d4988a4417-kube-api-access-d2kgq\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:57.904473 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904454    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904618 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904483    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904618 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904536    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:57.904618 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904575    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdb98e3-4600-4db4-9ab9-54d4988a4417-tmp\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:57.904753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904619    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vk2\" (UniqueName: \"kubernetes.io/projected/1ced107c-3de2-4c35-abe8-782f6d26ecc7-kube-api-access-q8vk2\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:57.904753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904652    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ced107c-3de2-4c35-abe8-782f6d26ecc7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:57.904753 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.904693    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5cdb98e3-4600-4db4-9ab9-54d4988a4417-klusterlet-config\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:57.949680 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.949646    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"]
Sep 08 16:51:57.952391 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.952374    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:57.967560 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:57.967536    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"]
Sep 08 16:51:58.005830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005796    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vk2\" (UniqueName: \"kubernetes.io/projected/1ced107c-3de2-4c35-abe8-782f6d26ecc7-kube-api-access-q8vk2\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:58.005830 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005832    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-bound-sa-token\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005853    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ced107c-3de2-4c35-abe8-782f6d26ecc7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:58.006018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005871    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-trusted-ca\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005921    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5cdb98e3-4600-4db4-9ab9-54d4988a4417-klusterlet-config\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.006018 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.005961    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs857\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-kube-api-access-gs857\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006162 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006020    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-tls\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006162 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006085    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e66ff-e961-4503-9752-b506feeeb46c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006162 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006114    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjd4\" (UniqueName: \"kubernetes.io/projected/979e66ff-e961-4503-9752-b506feeeb46c-kube-api-access-ffjd4\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006162 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006141    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-image-registry-private-configuration\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006384 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006165    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006384 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006347    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kgq\" (UniqueName: \"kubernetes.io/projected/5cdb98e3-4600-4db4-9ab9-54d4988a4417-kube-api-access-d2kgq\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.006480 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006388    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006480 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006430    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006480 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006461    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-certificates\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006630 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006520    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.006630 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006546    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-installation-pull-secrets\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006630 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006575    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5807b69-c876-46b5-84c8-1ac5e5336805-ca-trust-extracted\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.006630 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006614    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdb98e3-4600-4db4-9ab9-54d4988a4417-tmp\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.006845 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006824    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/979e66ff-e961-4503-9752-b506feeeb46c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.007022 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.006992    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cdb98e3-4600-4db4-9ab9-54d4988a4417-tmp\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.009012 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.008984    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-ca\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.009113 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.009087    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ced107c-3de2-4c35-abe8-782f6d26ecc7-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:58.009113 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.009106    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5cdb98e3-4600-4db4-9ab9-54d4988a4417-klusterlet-config\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.009275 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.009144    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.009625 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.009597    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.009705 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.009604    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/979e66ff-e961-4503-9752-b506feeeb46c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.015889 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.015864    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kgq\" (UniqueName: \"kubernetes.io/projected/5cdb98e3-4600-4db4-9ab9-54d4988a4417-kube-api-access-d2kgq\") pod \"klusterlet-addon-workmgr-666989dd7d-vtrc2\" (UID: \"5cdb98e3-4600-4db4-9ab9-54d4988a4417\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.015984 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.015904    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjd4\" (UniqueName: \"kubernetes.io/projected/979e66ff-e961-4503-9752-b506feeeb46c-kube-api-access-ffjd4\") pod \"cluster-proxy-proxy-agent-d4fff956-4sv56\" (UID: \"979e66ff-e961-4503-9752-b506feeeb46c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.016688 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.016668    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vk2\" (UniqueName: \"kubernetes.io/projected/1ced107c-3de2-4c35-abe8-782f6d26ecc7-kube-api-access-q8vk2\") pod \"managed-serviceaccount-addon-agent-76844f4d4-bfwhz\" (UID: \"1ced107c-3de2-4c35-abe8-782f6d26ecc7\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:58.107603 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107508    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-certificates\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107603 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107568    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-installation-pull-secrets\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107791 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107594    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5807b69-c876-46b5-84c8-1ac5e5336805-ca-trust-extracted\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107791 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107655    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-bound-sa-token\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107791 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107712    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-trusted-ca\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107791 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107739    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs857\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-kube-api-access-gs857\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107972 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107791    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-tls\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.107972 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.107837    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-image-registry-private-configuration\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.108110 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.108078    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c5807b69-c876-46b5-84c8-1ac5e5336805-ca-trust-extracted\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.108666 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.108643    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-trusted-ca\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.108791 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.108695    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-certificates\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.110133 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.110111    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-image-registry-private-configuration\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.110252 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.110228    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c5807b69-c876-46b5-84c8-1ac5e5336805-installation-pull-secrets\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.110394 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.110379    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-registry-tls\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.116568 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.116540    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs857\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-kube-api-access-gs857\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.116694 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.116677    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c5807b69-c876-46b5-84c8-1ac5e5336805-bound-sa-token\") pod \"image-registry-7d6ffbb4b-zxzkk\" (UID: \"c5807b69-c876-46b5-84c8-1ac5e5336805\") " pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.134195 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.134168    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"
Sep 08 16:51:58.140272 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.140251    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"
Sep 08 16:51:58.152667 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.152645    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:51:58.264157 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.264126    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f9rmv\""
Sep 08 16:51:58.271402 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.271378    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:58.396531 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.396451    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:58.412390 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.412346    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz"]
Sep 08 16:51:58.417906 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:58.417879    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ced107c_3de2_4c35_abe8_782f6d26ecc7.slice/crio-5a8b49329230ff744b02359536062710c80fcca6f437be99e50f59026699b822 WatchSource:0}: Error finding container 5a8b49329230ff744b02359536062710c80fcca6f437be99e50f59026699b822: Status 404 returned error can't find the container with id 5a8b49329230ff744b02359536062710c80fcca6f437be99e50f59026699b822
Sep 08 16:51:58.421878 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.421853    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:58.428779 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.428700    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56"]
Sep 08 16:51:58.431136 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:58.431115    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod979e66ff_e961_4503_9752_b506feeeb46c.slice/crio-c9188b5bc1686abceb99f7de65766f369d6fcc9e32bb5b48d832c6989e888105 WatchSource:0}: Error finding container c9188b5bc1686abceb99f7de65766f369d6fcc9e32bb5b48d832c6989e888105: Status 404 returned error can't find the container with id c9188b5bc1686abceb99f7de65766f369d6fcc9e32bb5b48d832c6989e888105
Sep 08 16:51:58.441360 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.441345    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"]
Sep 08 16:51:58.443626 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:58.443606    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdb98e3_4600_4db4_9ab9_54d4988a4417.slice/crio-a9a950db75ebf2a5379e20999dfd69682733bf1101f20981ad8bc4039d09feba WatchSource:0}: Error finding container a9a950db75ebf2a5379e20999dfd69682733bf1101f20981ad8bc4039d09feba: Status 404 returned error can't find the container with id a9a950db75ebf2a5379e20999dfd69682733bf1101f20981ad8bc4039d09feba
Sep 08 16:51:58.511256 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511230    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511443 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511272    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511443 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511314    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511443 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511398    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w79d7\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511585 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511528    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511585 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511569    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511686 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511600    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:58.511733 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511599    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:51:58.511774 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511738    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:58.511986 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511936    2542 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24ec4a68-9201-4c2f-927d-5901d2aa8a85-ca-trust-extracted\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.511986 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.511961    2542 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-certificates\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.512239 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.512219    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:51:58.513786 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.513750    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:58.513869 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.513817    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:58.513955 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.513938    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7" (OuterVolumeSpecName: "kube-api-access-w79d7") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "kube-api-access-w79d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:58.513995 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.513947    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:51:58.557929 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.557907    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"]
Sep 08 16:51:58.560041 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:58.560020    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5807b69_c876_46b5_84c8_1ac5e5336805.slice/crio-05d32bf57296cd7fc45bd44863fecbcfe05f64a2d991bab236ee032ec3835e06 WatchSource:0}: Error finding container 05d32bf57296cd7fc45bd44863fecbcfe05f64a2d991bab236ee032ec3835e06: Status 404 returned error can't find the container with id 05d32bf57296cd7fc45bd44863fecbcfe05f64a2d991bab236ee032ec3835e06
Sep 08 16:51:58.612989 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.612961    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79d7\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-kube-api-access-w79d7\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.612989 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.612988    2542 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-bound-sa-token\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.613131 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.612998    2542 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24ec4a68-9201-4c2f-927d-5901d2aa8a85-trusted-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.613131 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.613006    2542 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-installation-pull-secrets\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:58.613131 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:58.613015    2542 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24ec4a68-9201-4c2f-927d-5901d2aa8a85-image-registry-private-configuration\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:59.217634 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.217603    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:59.218026 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.217758    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:59.220094 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.220071    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30ac2086-1fcf-4ec7-9ed2-8202a55eb804-metrics-tls\") pod \"dns-default-6hvpz\" (UID: \"30ac2086-1fcf-4ec7-9ed2-8202a55eb804\") " pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:59.220282 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.220257    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"image-registry-569c94b8d6-pchlf\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") " pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:59.318486 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.318405    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") pod \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\" (UID: \"24ec4a68-9201-4c2f-927d-5901d2aa8a85\") "
Sep 08 16:51:59.320635 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.320603    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "24ec4a68-9201-4c2f-927d-5901d2aa8a85" (UID: "24ec4a68-9201-4c2f-927d-5901d2aa8a85"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:51:59.402234 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.402192    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56" event={"ID":"979e66ff-e961-4503-9752-b506feeeb46c","Type":"ContainerStarted","Data":"c9188b5bc1686abceb99f7de65766f369d6fcc9e32bb5b48d832c6989e888105"}
Sep 08 16:51:59.403647 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.403610    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz" event={"ID":"1ced107c-3de2-4c35-abe8-782f6d26ecc7","Type":"ContainerStarted","Data":"5a8b49329230ff744b02359536062710c80fcca6f437be99e50f59026699b822"}
Sep 08 16:51:59.405470 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.405441    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk" event={"ID":"c5807b69-c876-46b5-84c8-1ac5e5336805","Type":"ContainerStarted","Data":"17f20847194f652f8a818aad97f7bab28eb25e6b64d33bb0967879f0f4bd17a7"}
Sep 08 16:51:59.405613 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.405495    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk" event={"ID":"c5807b69-c876-46b5-84c8-1ac5e5336805","Type":"ContainerStarted","Data":"05d32bf57296cd7fc45bd44863fecbcfe05f64a2d991bab236ee032ec3835e06"}
Sep 08 16:51:59.405613 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.405523    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:51:59.407574 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.407474    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-569c94b8d6-pchlf"
Sep 08 16:51:59.407574 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.407485    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2" event={"ID":"5cdb98e3-4600-4db4-9ab9-54d4988a4417","Type":"ContainerStarted","Data":"a9a950db75ebf2a5379e20999dfd69682733bf1101f20981ad8bc4039d09feba"}
Sep 08 16:51:59.420961 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.420587    2542 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24ec4a68-9201-4c2f-927d-5901d2aa8a85-registry-tls\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 16:51:59.425727 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.425377    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk" podStartSLOduration=2.425360343 podStartE2EDuration="2.425360343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:59.424958177 +0000 UTC m=+62.892164370" watchObservedRunningTime="2026-09-08 16:51:59.425360343 +0000 UTC m=+62.892566537"
Sep 08 16:51:59.455662 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.455614    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-569c94b8d6-pchlf"]
Sep 08 16:51:59.456368 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.456166    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpm9l\""
Sep 08 16:51:59.458498 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.458453    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-569c94b8d6-pchlf"]
Sep 08 16:51:59.464574 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.464545    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:51:59.781839 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:51:59.780945    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hvpz"]
Sep 08 16:51:59.785754 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:51:59.785710    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ac2086_1fcf_4ec7_9ed2_8202a55eb804.slice/crio-3af65da190b3de56fe2b4257606911c7baa7a1667f04970574b4fc0560e360eb WatchSource:0}: Error finding container 3af65da190b3de56fe2b4257606911c7baa7a1667f04970574b4fc0560e360eb: Status 404 returned error can't find the container with id 3af65da190b3de56fe2b4257606911c7baa7a1667f04970574b4fc0560e360eb
Sep 08 16:52:00.415170 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:00.415136    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hvpz" event={"ID":"30ac2086-1fcf-4ec7-9ed2-8202a55eb804","Type":"ContainerStarted","Data":"3af65da190b3de56fe2b4257606911c7baa7a1667f04970574b4fc0560e360eb"}
Sep 08 16:52:01.116718 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:01.116464    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ec4a68-9201-4c2f-927d-5901d2aa8a85" path="/var/lib/kubelet/pods/24ec4a68-9201-4c2f-927d-5901d2aa8a85/volumes"
Sep 08 16:52:01.842884 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:01.842739    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:52:01.846087 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:01.845780    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdd679a3-1c44-40bc-9c88-f97be23eec57-metrics-certs\") pod \"network-metrics-daemon-2rxnz\" (UID: \"bdd679a3-1c44-40bc-9c88-f97be23eec57\") " pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:52:02.131690 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:02.131603    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wrtqz\""
Sep 08 16:52:02.139085 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:02.139057    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rxnz"
Sep 08 16:52:03.154302 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:03.154261    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:52:03.157116 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:03.157086    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6478c3cb-d09f-4aa6-bad7-5c87a320b03e-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cwjkc\" (UID: \"6478c3cb-d09f-4aa6-bad7-5c87a320b03e\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:52:03.391485 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:03.391450    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-cvbpd\""
Sep 08 16:52:03.398937 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:03.398906    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc"
Sep 08 16:52:04.769251 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:04.769204    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:52:04.771811 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:04.771789    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/36c70a95-9295-4368-a484-1c6dacf4d6e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k99cv\" (UID: \"36c70a95-9295-4368-a484-1c6dacf4d6e5\") " pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:52:05.025555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.025457    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxh7j\""
Sep 08 16:52:05.033714 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.033678    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k99cv"
Sep 08 16:52:05.444596 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.444562    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2" event={"ID":"5cdb98e3-4600-4db4-9ab9-54d4988a4417","Type":"ContainerStarted","Data":"8f57c283bc5aa23072a0b46aa84e291260c5327c997b36f1b572bbab7730a66d"}
Sep 08 16:52:05.446541 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.446508    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56" event={"ID":"979e66ff-e961-4503-9752-b506feeeb46c","Type":"ContainerStarted","Data":"081dbacc21f87fb5956d463c77790b74429d20d436ffadd2267820fa5da40d3c"}
Sep 08 16:52:05.448757 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.448723    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz" event={"ID":"1ced107c-3de2-4c35-abe8-782f6d26ecc7","Type":"ContainerStarted","Data":"def606f9129cd26b30c892d27ca608d70791e14e937cddd2a53814a2580d17b9"}
Sep 08 16:52:05.463131 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.462924    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:52:05.465771 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.465583    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2"
Sep 08 16:52:05.480305 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.480267    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-666989dd7d-vtrc2" podStartSLOduration=1.640647858 podStartE2EDuration="8.480252095s" totalImagesPullingTime="6.839604237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:05.479051524 +0000 UTC m=+68.946257739" watchObservedRunningTime="2026-09-08 16:52:05.480252095 +0000 UTC m=+68.947458289"
Sep 08 16:52:05.561285 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.561260    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cwjkc"]
Sep 08 16:52:05.566470 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:52:05.566397    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6478c3cb_d09f_4aa6_bad7_5c87a320b03e.slice/crio-a8afd21de035d5749d23f0b5abbfb5e5f758118d58bbe72be0db34bbaadaa17b WatchSource:0}: Error finding container a8afd21de035d5749d23f0b5abbfb5e5f758118d58bbe72be0db34bbaadaa17b: Status 404 returned error can't find the container with id a8afd21de035d5749d23f0b5abbfb5e5f758118d58bbe72be0db34bbaadaa17b
Sep 08 16:52:05.598832 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.598807    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2rxnz"]
Sep 08 16:52:05.600437 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:05.600410    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k99cv"]
Sep 08 16:52:05.604081 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:52:05.604034    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd679a3_1c44_40bc_9c88_f97be23eec57.slice/crio-88424ad2f2c0f1abfb8ddcee38356f748dc68e950238beb6b6552832c5ab3c98 WatchSource:0}: Error finding container 88424ad2f2c0f1abfb8ddcee38356f748dc68e950238beb6b6552832c5ab3c98: Status 404 returned error can't find the container with id 88424ad2f2c0f1abfb8ddcee38356f748dc68e950238beb6b6552832c5ab3c98
Sep 08 16:52:06.454250 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.454212    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99cv" event={"ID":"36c70a95-9295-4368-a484-1c6dacf4d6e5","Type":"ContainerStarted","Data":"4c27758dff70e8ba32ef803a968fc826e0b461f22b82deb3879cbe139f5d40b3"}
Sep 08 16:52:06.454649 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.454341    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99cv" event={"ID":"36c70a95-9295-4368-a484-1c6dacf4d6e5","Type":"ContainerStarted","Data":"e0319d4877809d032627634e0428f3ff21f5900e7f914020836d2f5b00aad746"}
Sep 08 16:52:06.455300 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.455282    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rxnz" event={"ID":"bdd679a3-1c44-40bc-9c88-f97be23eec57","Type":"ContainerStarted","Data":"88424ad2f2c0f1abfb8ddcee38356f748dc68e950238beb6b6552832c5ab3c98"}
Sep 08 16:52:06.456348 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.456312    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc" event={"ID":"6478c3cb-d09f-4aa6-bad7-5c87a320b03e","Type":"ContainerStarted","Data":"a8afd21de035d5749d23f0b5abbfb5e5f758118d58bbe72be0db34bbaadaa17b"}
Sep 08 16:52:06.457872 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.457852    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hvpz" event={"ID":"30ac2086-1fcf-4ec7-9ed2-8202a55eb804","Type":"ContainerStarted","Data":"1c58de6967a364b997ca7d4ca4c9acab9516ab982ddde3e9e3af8d9de40792b6"}
Sep 08 16:52:06.457951 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.457889    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hvpz" event={"ID":"30ac2086-1fcf-4ec7-9ed2-8202a55eb804","Type":"ContainerStarted","Data":"faee715b161ebb1660c57f49af30779e6789d0143017c8db04e562c44fa75c7d"}
Sep 08 16:52:06.457994 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.457965    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:52:06.459834 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.459814    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56" event={"ID":"979e66ff-e961-4503-9752-b506feeeb46c","Type":"ContainerStarted","Data":"c9563b241668cf7202263a37829d4b61268f5cf245d59490b7f8b3b38df21fef"}
Sep 08 16:52:06.459896 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.459845    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56" event={"ID":"979e66ff-e961-4503-9752-b506feeeb46c","Type":"ContainerStarted","Data":"b8eeb1e39088267d0b41a20e517a7333577a1208e41949ece1a2f64c60eaad81"}
Sep 08 16:52:06.476096 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.476056    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6hvpz" podStartSLOduration=33.969082061 podStartE2EDuration="39.476041141s" totalImagesPullingTime="5.50695908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:06.474931841 +0000 UTC m=+69.942138034" watchObservedRunningTime="2026-09-08 16:52:06.476041141 +0000 UTC m=+69.943247333"
Sep 08 16:52:06.489829 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.489794    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76844f4d4-bfwhz" podStartSLOduration=2.617160194 podStartE2EDuration="9.489779743s" totalImagesPullingTime="6.872619549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:06.489778186 +0000 UTC m=+69.956984373" watchObservedRunningTime="2026-09-08 16:52:06.489779743 +0000 UTC m=+69.956985933"
Sep 08 16:52:06.506666 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:06.506622    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fff956-4sv56" podStartSLOduration=2.64989564 podStartE2EDuration="9.506609151s" totalImagesPullingTime="6.856713511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:06.506304833 +0000 UTC m=+69.973511025" watchObservedRunningTime="2026-09-08 16:52:06.506609151 +0000 UTC m=+69.973815344"
Sep 08 16:52:08.322091 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.322061    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6vkpg"
Sep 08 16:52:08.470787 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.470750    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99cv" event={"ID":"36c70a95-9295-4368-a484-1c6dacf4d6e5","Type":"ContainerStarted","Data":"888124da5f2c2adff05d69ff1c3217e71d8f3b649032def9792e64833e52822e"}
Sep 08 16:52:08.472536 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.472513    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rxnz" event={"ID":"bdd679a3-1c44-40bc-9c88-f97be23eec57","Type":"ContainerStarted","Data":"0b5101c53170eece148293239a486e44c5985eaa5c6b3d0443d781bffb7568c7"}
Sep 08 16:52:08.472680 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.472552    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rxnz" event={"ID":"bdd679a3-1c44-40bc-9c88-f97be23eec57","Type":"ContainerStarted","Data":"f935cf5b3314900852695b47fb8a59de92249e21a86a3e5e5808ae7af980e0c0"}
Sep 08 16:52:08.474377 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.474354    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc" event={"ID":"6478c3cb-d09f-4aa6-bad7-5c87a320b03e","Type":"ContainerStarted","Data":"1edbf72e01853fc495a9ca97a8bf357b46662947014ca2bdd8538d76e9d2457c"}
Sep 08 16:52:08.530501 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.530448    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2rxnz" podStartSLOduration=69.06148354 podStartE2EDuration="1m11.530431462s" totalImagesPullingTime="2.468947922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:08.498753361 +0000 UTC m=+71.965959554" watchObservedRunningTime="2026-09-08 16:52:08.530431462 +0000 UTC m=+71.997637803"
Sep 08 16:52:08.530660 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:08.530600    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-cwjkc" podStartSLOduration=35.025552861 podStartE2EDuration="37.530592496s" totalImagesPullingTime="2.505039635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:08.529111817 +0000 UTC m=+71.996318010" watchObservedRunningTime="2026-09-08 16:52:08.530592496 +0000 UTC m=+71.997798688"
Sep 08 16:52:11.229698 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.229666    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qnhg8"]
Sep 08 16:52:11.249411 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.249380    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.252221 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252195    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 16:52:11.252406 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252251    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 16:52:11.252406 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252377    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-crsks\""
Sep 08 16:52:11.252534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252414    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 16:52:11.252534 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252527    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 16:52:11.252680 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.252642    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 16:52:11.253275 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.253260    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325274    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-root\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325394    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-metrics-client-ca\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325487    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-tls\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325525    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zh2\" (UniqueName: \"kubernetes.io/projected/2a483673-11bf-44ec-9286-4a6c6a483a39-kube-api-access-b5zh2\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325593    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325634    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-sys\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325666    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-wtmp\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325719    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-textfile\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.329066 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.325766    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427183 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427138    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427191    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-sys\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427219    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-wtmp\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427263    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-textfile\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427298    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427348    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-sys\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427381 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427358    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-root\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427677 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427405    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-root\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427677 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427419    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-metrics-client-ca\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427677 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427450    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-tls\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427677 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427451    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-wtmp\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427677 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427484    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zh2\" (UniqueName: \"kubernetes.io/projected/2a483673-11bf-44ec-9286-4a6c6a483a39-kube-api-access-b5zh2\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.427939 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.427733    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-textfile\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.428041 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.428015    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.428159 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.428144    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2a483673-11bf-44ec-9286-4a6c6a483a39-metrics-client-ca\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.429546 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.429523    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.429648 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.429623    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2a483673-11bf-44ec-9286-4a6c6a483a39-node-exporter-tls\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.436088 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.436063    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zh2\" (UniqueName: \"kubernetes.io/projected/2a483673-11bf-44ec-9286-4a6c6a483a39-kube-api-access-b5zh2\") pod \"node-exporter-qnhg8\" (UID: \"2a483673-11bf-44ec-9286-4a6c6a483a39\") " pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.486170 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.486136    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k99cv" event={"ID":"36c70a95-9295-4368-a484-1c6dacf4d6e5","Type":"ContainerStarted","Data":"7f97afb920bf3d544618a1a870cdcc557c5942384c7f2e83e97250cd86cd4a16"}
Sep 08 16:52:11.506127 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.506075    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k99cv" podStartSLOduration=34.371023939 podStartE2EDuration="39.506057207s" totalImagesPullingTime="5.135033268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:11.505088759 +0000 UTC m=+74.972294953" watchObservedRunningTime="2026-09-08 16:52:11.506057207 +0000 UTC m=+74.973263401"
Sep 08 16:52:11.558886 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:11.558857    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qnhg8"
Sep 08 16:52:11.599862 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:52:11.599816    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a483673_11bf_44ec_9286_4a6c6a483a39.slice/crio-019ad67b3830c2dd28da1005b4e98228077a2ef2074969cc24eef3ec101e4a8a WatchSource:0}: Error finding container 019ad67b3830c2dd28da1005b4e98228077a2ef2074969cc24eef3ec101e4a8a: Status 404 returned error can't find the container with id 019ad67b3830c2dd28da1005b4e98228077a2ef2074969cc24eef3ec101e4a8a
Sep 08 16:52:12.491295 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:12.491256    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnhg8" event={"ID":"2a483673-11bf-44ec-9286-4a6c6a483a39","Type":"ContainerStarted","Data":"019ad67b3830c2dd28da1005b4e98228077a2ef2074969cc24eef3ec101e4a8a"}
Sep 08 16:52:14.503272 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:14.503237    2542 generic.go:356] "Generic (PLEG): container finished" podID="2a483673-11bf-44ec-9286-4a6c6a483a39" containerID="40e9f5e290e84173a44ca05f18cb2bd421ba1cac2b9748e20db402f3029faa7a" exitCode=0
Sep 08 16:52:14.503272 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:14.503276    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnhg8" event={"ID":"2a483673-11bf-44ec-9286-4a6c6a483a39","Type":"ContainerDied","Data":"40e9f5e290e84173a44ca05f18cb2bd421ba1cac2b9748e20db402f3029faa7a"}
Sep 08 16:52:15.508495 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:15.508454    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnhg8" event={"ID":"2a483673-11bf-44ec-9286-4a6c6a483a39","Type":"ContainerStarted","Data":"3bee268c14670c721bd7bb00cbbe978c453ea5903c797877581dd742b012d886"}
Sep 08 16:52:15.508845 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:15.508505    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qnhg8" event={"ID":"2a483673-11bf-44ec-9286-4a6c6a483a39","Type":"ContainerStarted","Data":"e632fb79c745ee8699b64fa8e5408e7e2cb2fd5da0983974dedf6c1c17c6d7cf"}
Sep 08 16:52:16.465894 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:16.465862    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6hvpz"
Sep 08 16:52:16.492946 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:16.492885    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qnhg8" podStartSLOduration=3.445253505 podStartE2EDuration="5.492865539s" totalImagesPullingTime="1.96970511s" totalInitContainerRuntime="77.906924ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:15.543790178 +0000 UTC m=+79.010996372" watchObservedRunningTime="2026-09-08 16:52:16.492865539 +0000 UTC m=+79.960071766"
Sep 08 16:52:18.276483 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:18.276447    2542 patch_prober.go:28] interesting pod/image-registry-7d6ffbb4b-zxzkk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:52:18.276854 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:18.276512    2542 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk" podUID="c5807b69-c876-46b5-84c8-1ac5e5336805" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:52:20.419556 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:52:20.419446    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d6ffbb4b-zxzkk"
Sep 08 16:55:24.279590 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.279510    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"]
Sep 08 16:55:24.281509 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.281493    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.284555 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.284537    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 16:55:24.284834 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.284814    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l98dr\""
Sep 08 16:55:24.285559 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.285541    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 16:55:24.285836 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.285822    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 16:55:24.325193 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.325164    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 16:55:24.353752 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.353725    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"]
Sep 08 16:55:24.453116 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.453081    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.453299 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.453151    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742wc\" (UniqueName: \"kubernetes.io/projected/7ec0db90-7fd4-43d9-99e0-714138427a4a-kube-api-access-742wc\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.453299 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.453179    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.453299 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.453199    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.453299 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.453219    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554097 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554006    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554097 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554049    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554097 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554069    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554317 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554195    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554317 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554302    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742wc\" (UniqueName: \"kubernetes.io/projected/7ec0db90-7fd4-43d9-99e0-714138427a4a-kube-api-access-742wc\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.554494 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.554475    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-manifests\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.556608 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.556584    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ec0db90-7fd4-43d9-99e0-714138427a4a-tmp\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.556750 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.556701    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.556865 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.556845    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ec0db90-7fd4-43d9-99e0-714138427a4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.569306 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.569285    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-742wc\" (UniqueName: \"kubernetes.io/projected/7ec0db90-7fd4-43d9-99e0-714138427a4a-kube-api-access-742wc\") pod \"opendatahub-operator-controller-manager-84f58c54b-lxtjp\" (UID: \"7ec0db90-7fd4-43d9-99e0-714138427a4a\") " pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.591439 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.591412    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:24.908221 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:24.908148    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"]
Sep 08 16:55:24.911460 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:55:24.911432    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec0db90_7fd4_43d9_99e0_714138427a4a.slice/crio-124b0f3176a9495187611cbae793079a1b65dcc96caa8fd93af61c8fe82b22c3 WatchSource:0}: Error finding container 124b0f3176a9495187611cbae793079a1b65dcc96caa8fd93af61c8fe82b22c3: Status 404 returned error can't find the container with id 124b0f3176a9495187611cbae793079a1b65dcc96caa8fd93af61c8fe82b22c3
Sep 08 16:55:25.159205 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:25.159120    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp" event={"ID":"7ec0db90-7fd4-43d9-99e0-714138427a4a","Type":"ContainerStarted","Data":"124b0f3176a9495187611cbae793079a1b65dcc96caa8fd93af61c8fe82b22c3"}
Sep 08 16:55:28.176668 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:28.176628    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp" event={"ID":"7ec0db90-7fd4-43d9-99e0-714138427a4a","Type":"ContainerStarted","Data":"56dd7904dfc585669b1ea2014683d3ab91676d2552ebe8337b90c18f91a4c180"}
Sep 08 16:55:29.182262 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:29.182225    2542 generic.go:356] "Generic (PLEG): container finished" podID="7ec0db90-7fd4-43d9-99e0-714138427a4a" containerID="56dd7904dfc585669b1ea2014683d3ab91676d2552ebe8337b90c18f91a4c180" exitCode=0
Sep 08 16:55:29.182661 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:29.182292    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp" event={"ID":"7ec0db90-7fd4-43d9-99e0-714138427a4a","Type":"ContainerDied","Data":"56dd7904dfc585669b1ea2014683d3ab91676d2552ebe8337b90c18f91a4c180"}
Sep 08 16:55:30.188194 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:30.188160    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp" event={"ID":"7ec0db90-7fd4-43d9-99e0-714138427a4a","Type":"ContainerStarted","Data":"7757334f91cfb557d1e458a8fcba2e104002dc521f523294573e7f6255e38b6d"}
Sep 08 16:55:30.188625 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:30.188347    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:30.301174 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:30.301116    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp" podStartSLOduration=1.648236164 podStartE2EDuration="6.301099109s" totalImagesPullingTime="3.657214835s" totalInitContainerRuntime="995.64811ms" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:30.298744752 +0000 UTC m=+273.765950946" watchObservedRunningTime="2026-09-08 16:55:30.301099109 +0000 UTC m=+273.768305302"
Sep 08 16:55:41.195125 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:41.195092    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84f58c54b-lxtjp"
Sep 08 16:55:45.614172 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.614134    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"]
Sep 08 16:55:45.623576 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.623550    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.631344 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.631199    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 16:55:45.631344 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.631203    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rchrb\""
Sep 08 16:55:45.631344 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.631245    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 16:55:45.632599 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.632576    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"]
Sep 08 16:55:45.632955 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.632931    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 16:55:45.656268 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.656236    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 16:55:45.715285 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.715252    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.715285 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.715292    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tmp\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.715516 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.715318    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbnn\" (UniqueName: \"kubernetes.io/projected/83282811-8ca5-4e9d-93f8-03277ff5d0c8-kube-api-access-zfbnn\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.815948 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.815914    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.815948 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.815961    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tmp\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.816212 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.815997    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbnn\" (UniqueName: \"kubernetes.io/projected/83282811-8ca5-4e9d-93f8-03277ff5d0c8-kube-api-access-zfbnn\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.818274 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.818245    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tmp\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.818508 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.818490    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83282811-8ca5-4e9d-93f8-03277ff5d0c8-tls-certs\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.871142 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.871067    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbnn\" (UniqueName: \"kubernetes.io/projected/83282811-8ca5-4e9d-93f8-03277ff5d0c8-kube-api-access-zfbnn\") pod \"kube-auth-proxy-7d8677cc5f-vslwz\" (UID: \"83282811-8ca5-4e9d-93f8-03277ff5d0c8\") " pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:45.934125 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:45.934088    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"
Sep 08 16:55:46.243507 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:46.243475    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz"]
Sep 08 16:55:46.246683 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:55:46.246659    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83282811_8ca5_4e9d_93f8_03277ff5d0c8.slice/crio-8f4077e269a630a46bd54d0338430547df704575e129c7d28320184aaaf9b68c WatchSource:0}: Error finding container 8f4077e269a630a46bd54d0338430547df704575e129c7d28320184aaaf9b68c: Status 404 returned error can't find the container with id 8f4077e269a630a46bd54d0338430547df704575e129c7d28320184aaaf9b68c
Sep 08 16:55:47.248033 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:47.247992    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz" event={"ID":"83282811-8ca5-4e9d-93f8-03277ff5d0c8","Type":"ContainerStarted","Data":"8f4077e269a630a46bd54d0338430547df704575e129c7d28320184aaaf9b68c"}
Sep 08 16:55:49.257079 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:49.257045    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz" event={"ID":"83282811-8ca5-4e9d-93f8-03277ff5d0c8","Type":"ContainerStarted","Data":"bbe6748e2d60bf1dc879c33ef6103ed4bb017c553d548470117dc2bceb7277e6"}
Sep 08 16:55:49.301954 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:49.301897    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d8677cc5f-vslwz" podStartSLOduration=1.637696209 podStartE2EDuration="4.301878481s" totalImagesPullingTime="2.664182272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:55:49.300008226 +0000 UTC m=+292.767214419" watchObservedRunningTime="2026-09-08 16:55:49.301878481 +0000 UTC m=+292.769084673"
Sep 08 16:55:55.686005 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.685970    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-68hh5"]
Sep 08 16:55:55.693881 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.693849    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:55.697004 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.696976    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mv58t\""
Sep 08 16:55:55.711917 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.711885    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-68hh5"]
Sep 08 16:55:55.739222 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.739189    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 08 16:55:55.795247 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.795210    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnvg\" (UniqueName: \"kubernetes.io/projected/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-kube-api-access-tvnvg\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:55.795439 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.795275    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:55.895977 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.895940    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:55.896138 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.895997    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnvg\" (UniqueName: \"kubernetes.io/projected/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-kube-api-access-tvnvg\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:55.896138 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:55:55.896084    2542 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 08 16:55:55.896207 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:55:55.896156    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert podName:fd0a5bf0-7fe5-47aa-b848-73c84cc71f79 nodeName:}" failed. No retries permitted until 2026-09-08 16:55:56.396138687 +0000 UTC m=+299.863344859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert") pod "odh-model-controller-7998c67dc4-68hh5" (UID: "fd0a5bf0-7fe5-47aa-b848-73c84cc71f79") : secret "odh-model-controller-webhook-cert" not found
Sep 08 16:55:55.965826 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:55.965742    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnvg\" (UniqueName: \"kubernetes.io/projected/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-kube-api-access-tvnvg\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:56.399590 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.399497    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:56.401912 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.401887    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd0a5bf0-7fe5-47aa-b848-73c84cc71f79-cert\") pod \"odh-model-controller-7998c67dc4-68hh5\" (UID: \"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79\") " pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:56.604048 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.604014    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:55:56.914047 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.913926    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-68hh5"]
Sep 08 16:55:56.918772 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:55:56.918733    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd0a5bf0_7fe5_47aa_b848_73c84cc71f79.slice/crio-ecae515d7b11ad66e7c5792cd9353c32b1fe41c4d7872672a61f686331d0c3de WatchSource:0}: Error finding container ecae515d7b11ad66e7c5792cd9353c32b1fe41c4d7872672a61f686331d0c3de: Status 404 returned error can't find the container with id ecae515d7b11ad66e7c5792cd9353c32b1fe41c4d7872672a61f686331d0c3de
Sep 08 16:55:56.939528 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.939500    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 16:55:56.940424 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.940340    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 16:55:56.945542 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:56.945518    2542 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 16:55:57.010949 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.010914    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b28bb"]
Sep 08 16:55:57.016123 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.016001    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.025835 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.025817    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 08 16:55:57.042138 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.042111    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b28bb"]
Sep 08 16:55:57.105497 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.105462    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mrj\" (UniqueName: \"kubernetes.io/projected/95295e73-58df-4462-82be-e0f2617357fc-kube-api-access-g4mrj\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.105670 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.105531    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.206382 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.206344    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.206556 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.206419    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mrj\" (UniqueName: \"kubernetes.io/projected/95295e73-58df-4462-82be-e0f2617357fc-kube-api-access-g4mrj\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.206556 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:55:57.206494    2542 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 08 16:55:57.206627 ip-10-0-133-106 kubenswrapper[2542]: E0908 16:55:57.206557    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs podName:95295e73-58df-4462-82be-e0f2617357fc nodeName:}" failed. No retries permitted until 2026-09-08 16:55:57.706538707 +0000 UTC m=+301.173744900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs") pod "model-serving-api-54d89bd5fc-b28bb" (UID: "95295e73-58df-4462-82be-e0f2617357fc") : secret "model-serving-api-tls" not found
Sep 08 16:55:57.222958 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.222922    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mrj\" (UniqueName: \"kubernetes.io/projected/95295e73-58df-4462-82be-e0f2617357fc-kube-api-access-g4mrj\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.285904 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.285865    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5" event={"ID":"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79","Type":"ContainerStarted","Data":"ecae515d7b11ad66e7c5792cd9353c32b1fe41c4d7872672a61f686331d0c3de"}
Sep 08 16:55:57.711591 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.711528    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.717710 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.717675    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95295e73-58df-4462-82be-e0f2617357fc-tls-certs\") pod \"model-serving-api-54d89bd5fc-b28bb\" (UID: \"95295e73-58df-4462-82be-e0f2617357fc\") " pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:57.926119 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:57.926080    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:55:58.234344 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:58.233608    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b28bb"]
Sep 08 16:55:58.236728 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:55:58.236540    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95295e73_58df_4462_82be_e0f2617357fc.slice/crio-f9b89e805d19e8ee0b9d87c2f5d530f45ed84cd010c5d48bfc1a8b5061f44154 WatchSource:0}: Error finding container f9b89e805d19e8ee0b9d87c2f5d530f45ed84cd010c5d48bfc1a8b5061f44154: Status 404 returned error can't find the container with id f9b89e805d19e8ee0b9d87c2f5d530f45ed84cd010c5d48bfc1a8b5061f44154
Sep 08 16:55:58.291226 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:55:58.291186    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb" event={"ID":"95295e73-58df-4462-82be-e0f2617357fc","Type":"ContainerStarted","Data":"f9b89e805d19e8ee0b9d87c2f5d530f45ed84cd010c5d48bfc1a8b5061f44154"}
Sep 08 16:56:00.302072 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:00.302027    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5" event={"ID":"fd0a5bf0-7fe5-47aa-b848-73c84cc71f79","Type":"ContainerStarted","Data":"5f5dcb53b1fc3c3b4d060452f4b007506d2ca7be109c340e706d24a18bf4658a"}
Sep 08 16:56:00.302573 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:00.302193    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:56:00.331672 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:00.331611    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5" podStartSLOduration=2.578114684 podStartE2EDuration="5.331586724s" totalImagesPullingTime="2.75347204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:00.330471992 +0000 UTC m=+303.797678249" watchObservedRunningTime="2026-09-08 16:56:00.331586724 +0000 UTC m=+303.798792918"
Sep 08 16:56:01.306681 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:01.306643    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb" event={"ID":"95295e73-58df-4462-82be-e0f2617357fc","Type":"ContainerStarted","Data":"0958d439a09c5c8b96b699ad3067fd8495eaa4a9f0394bdd09faf8f9ec040232"}
Sep 08 16:56:01.307115 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:01.306866    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:56:01.330806 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:01.330758    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb" podStartSLOduration=2.824428372 podStartE2EDuration="5.330740877s" totalImagesPullingTime="2.506312505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:01.330076282 +0000 UTC m=+304.797282487" watchObservedRunningTime="2026-09-08 16:56:01.330740877 +0000 UTC m=+304.797947070"
Sep 08 16:56:03.486151 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.486113    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"]
Sep 08 16:56:03.489296 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.489280    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.497470 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.497450    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 16:56:03.497559 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.497450    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 16:56:03.498649 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.498634    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 16:56:03.503395 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.503378    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xfgpt\""
Sep 08 16:56:03.503683 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.503669    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:56:03.503860 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.503848    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 16:56:03.519401 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.519377    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"]
Sep 08 16:56:03.562801 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.562764    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.562988 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.562814    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34357644-0983-4721-8004-653ce1db3b05-manager-config\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.562988 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.562901    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.562988 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.562963    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s99\" (UniqueName: \"kubernetes.io/projected/34357644-0983-4721-8004-653ce1db3b05-kube-api-access-m6s99\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.663581 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.663534    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.663581 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.663603    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s99\" (UniqueName: \"kubernetes.io/projected/34357644-0983-4721-8004-653ce1db3b05-kube-api-access-m6s99\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.663859 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.663628    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.663859 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.663660    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34357644-0983-4721-8004-653ce1db3b05-manager-config\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.664309 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.664287    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34357644-0983-4721-8004-653ce1db3b05-manager-config\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.666089 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.666066    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.666192 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.666116    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34357644-0983-4721-8004-653ce1db3b05-metrics-cert\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.678415 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.678387    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s99\" (UniqueName: \"kubernetes.io/projected/34357644-0983-4721-8004-653ce1db3b05-kube-api-access-m6s99\") pod \"lws-controller-manager-74fc5c54f8-jgz68\" (UID: \"34357644-0983-4721-8004-653ce1db3b05\") " pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:03.798582 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:03.798497    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:04.118892 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:04.118777    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"]
Sep 08 16:56:04.120897 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:56:04.120866    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34357644_0983_4721_8004_653ce1db3b05.slice/crio-6cddee192e7c0c54aff5aaecf2178ecf073b83aa13413dbea1f25fc80faf59a3 WatchSource:0}: Error finding container 6cddee192e7c0c54aff5aaecf2178ecf073b83aa13413dbea1f25fc80faf59a3: Status 404 returned error can't find the container with id 6cddee192e7c0c54aff5aaecf2178ecf073b83aa13413dbea1f25fc80faf59a3
Sep 08 16:56:04.122903 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:04.122888    2542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:56:04.319923 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:04.319883    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68" event={"ID":"34357644-0983-4721-8004-653ce1db3b05","Type":"ContainerStarted","Data":"6cddee192e7c0c54aff5aaecf2178ecf073b83aa13413dbea1f25fc80faf59a3"}
Sep 08 16:56:07.339190 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:07.339151    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68" event={"ID":"34357644-0983-4721-8004-653ce1db3b05","Type":"ContainerStarted","Data":"0341ef767fd806721ee888ec8b8d124f2e72ae729469aacd8e879e113dd21bf8"}
Sep 08 16:56:07.339672 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:07.339241    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:56:07.385670 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:07.385615    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68" podStartSLOduration=1.8805201569999999 podStartE2EDuration="4.385598181s" totalImagesPullingTime="2.505078024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:56:07.384926014 +0000 UTC m=+310.852132207" watchObservedRunningTime="2026-09-08 16:56:07.385598181 +0000 UTC m=+310.852804375"
Sep 08 16:56:11.309114 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:11.309078    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-68hh5"
Sep 08 16:56:12.315183 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:12.315154    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-b28bb"
Sep 08 16:56:18.344807 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:56:18.344777    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74fc5c54f8-jgz68"
Sep 08 16:59:36.320641 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.320606    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"]
Sep 08 16:59:36.323615 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.323598    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.327623 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.327595    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 16:59:36.327737 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.327627    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-z9jf4\""
Sep 08 16:59:36.340367 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.340336    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"]
Sep 08 16:59:36.444803 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.444768    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5d6x\" (UniqueName: \"kubernetes.io/projected/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-kube-api-access-h5d6x\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.444975 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.444814    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.545186 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.545148    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5d6x\" (UniqueName: \"kubernetes.io/projected/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-kube-api-access-h5d6x\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.545399 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.545202    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.545977 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.545958    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.554065 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.554037    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5d6x\" (UniqueName: \"kubernetes.io/projected/d338767d-21e5-4f2e-aa1f-95beee4ad7c0-kube-api-access-h5d6x\") pod \"payload-pre-processing-59fbc47bf8-vww8c\" (UID: \"d338767d-21e5-4f2e-aa1f-95beee4ad7c0\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.633061 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.632970    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:36.907550 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:36.907460    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"]
Sep 08 16:59:36.910405 ip-10-0-133-106 kubenswrapper[2542]: W0908 16:59:36.910376    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd338767d_21e5_4f2e_aa1f_95beee4ad7c0.slice/crio-4e6c8e853aceba03bbdff9c1b6de6410eea322de9a2d5b09ba40c7ada1e6a30e WatchSource:0}: Error finding container 4e6c8e853aceba03bbdff9c1b6de6410eea322de9a2d5b09ba40c7ada1e6a30e: Status 404 returned error can't find the container with id 4e6c8e853aceba03bbdff9c1b6de6410eea322de9a2d5b09ba40c7ada1e6a30e
Sep 08 16:59:37.176194 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:37.176156    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c" event={"ID":"d338767d-21e5-4f2e-aa1f-95beee4ad7c0","Type":"ContainerStarted","Data":"4e6c8e853aceba03bbdff9c1b6de6410eea322de9a2d5b09ba40c7ada1e6a30e"}
Sep 08 16:59:40.191238 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:40.191203    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c" event={"ID":"d338767d-21e5-4f2e-aa1f-95beee4ad7c0","Type":"ContainerStarted","Data":"1cfa19e8fb8c12fc191a63d3c829af5bab970bc7e3cc3b864704f8cfe0417772"}
Sep 08 16:59:40.191683 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:40.191305    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 16:59:40.211611 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:40.211555    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c" podStartSLOduration=1.973492051 podStartE2EDuration="4.21153839s" totalImagesPullingTime="2.238046339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:59:40.210987189 +0000 UTC m=+523.678193381" watchObservedRunningTime="2026-09-08 16:59:40.21153839 +0000 UTC m=+523.678744583"
Sep 08 16:59:51.199179 ip-10-0-133-106 kubenswrapper[2542]: I0908 16:59:51.199100    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-vww8c"
Sep 08 17:00:00.138913 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.138877    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"]
Sep 08 17:00:00.141813 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.141795    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"
Sep 08 17:00:00.145017 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.144997    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:00:00.151145 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.151120    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"
Sep 08 17:00:00.152925 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.152902    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"]
Sep 08 17:00:00.426705 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:00.426678    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"]
Sep 08 17:00:00.429135 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:00:00.429110    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8a28ca_81a5_450b_bab2_9f5041a4cb23.slice/crio-2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb WatchSource:0}: Error finding container 2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb: Status 404 returned error can't find the container with id 2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb
Sep 08 17:00:01.281802 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:01.281767    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g" event={"ID":"bd8a28ca-81a5-450b-bab2-9f5041a4cb23","Type":"ContainerStarted","Data":"2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb"}
Sep 08 17:00:02.621976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.621934    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"]
Sep 08 17:00:02.625939 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.625923    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.629811 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.629786    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 17:00:02.629943 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.629893    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 17:00:02.631010 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.630985    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9fdqs\""
Sep 08 17:00:02.631111 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.631091    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 17:00:02.638668 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.638647    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"]
Sep 08 17:00:02.772281 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772241    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nwx\" (UniqueName: \"kubernetes.io/projected/90dcba84-5f18-49b2-bad6-511f49565d3c-kube-api-access-b5nwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.772466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772301    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.772466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772392    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcba84-5f18-49b2-bad6-511f49565d3c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.772466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772424    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.772466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772445    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.772601 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.772470    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873645 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873563    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873645 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873621    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcba84-5f18-49b2-bad6-511f49565d3c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873645 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873650    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873830 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873678    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873830 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873712    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.873830 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.873767    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nwx\" (UniqueName: \"kubernetes.io/projected/90dcba84-5f18-49b2-bad6-511f49565d3c-kube-api-access-b5nwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.874057 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.874036    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.874151 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.874094    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.874151 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.874107    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.875949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.875926    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90dcba84-5f18-49b2-bad6-511f49565d3c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.876187 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.876171    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90dcba84-5f18-49b2-bad6-511f49565d3c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.890063 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.890037    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nwx\" (UniqueName: \"kubernetes.io/projected/90dcba84-5f18-49b2-bad6-511f49565d3c-kube-api-access-b5nwx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g\" (UID: \"90dcba84-5f18-49b2-bad6-511f49565d3c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:02.936137 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:02.936098    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:03.214134 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:03.214100    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"]
Sep 08 17:00:03.216875 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:00:03.216848    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dcba84_5f18_49b2_bad6_511f49565d3c.slice/crio-5dd9249e55a8ebd61ff63a146fdc1d94c226aebb04337fa3d6e45eada6db115a WatchSource:0}: Error finding container 5dd9249e55a8ebd61ff63a146fdc1d94c226aebb04337fa3d6e45eada6db115a: Status 404 returned error can't find the container with id 5dd9249e55a8ebd61ff63a146fdc1d94c226aebb04337fa3d6e45eada6db115a
Sep 08 17:00:03.291229 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:03.291190    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g" event={"ID":"90dcba84-5f18-49b2-bad6-511f49565d3c","Type":"ContainerStarted","Data":"5dd9249e55a8ebd61ff63a146fdc1d94c226aebb04337fa3d6e45eada6db115a"}
Sep 08 17:00:06.307959 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:06.307926    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g" event={"ID":"90dcba84-5f18-49b2-bad6-511f49565d3c","Type":"ContainerStarted","Data":"abb507a742e3d38af385e9a0279fb25f3f7b8d2eebb68ee6d8a0d5b685c3f810"}
Sep 08 17:00:06.331064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:06.331011    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g" podStartSLOduration=1.843664314 podStartE2EDuration="4.330994994s" totalImagesPullingTime="2.48733068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:06.328998375 +0000 UTC m=+549.796204567" watchObservedRunningTime="2026-09-08 17:00:06.330994994 +0000 UTC m=+549.798201178"
Sep 08 17:00:07.316411 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:07.316370    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g" event={"ID":"bd8a28ca-81a5-450b-bab2-9f5041a4cb23","Type":"ContainerStarted","Data":"69625aa4d3bd8e68f0bee0072a8615ae22cfd9ef916154f58a9af85bee853334"}
Sep 08 17:00:07.334563 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:07.334501    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g" podStartSLOduration=1.142879794 podStartE2EDuration="7.334482848s" totalImagesPullingTime="6.191603054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:07.332560302 +0000 UTC m=+550.799766495" watchObservedRunningTime="2026-09-08 17:00:07.334482848 +0000 UTC m=+550.801689040"
Sep 08 17:00:08.322519 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:08.322483    2542 generic.go:356] "Generic (PLEG): container finished" podID="bd8a28ca-81a5-450b-bab2-9f5041a4cb23" containerID="69625aa4d3bd8e68f0bee0072a8615ae22cfd9ef916154f58a9af85bee853334" exitCode=0
Sep 08 17:00:08.322903 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:08.322573    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g" event={"ID":"bd8a28ca-81a5-450b-bab2-9f5041a4cb23","Type":"ContainerDied","Data":"69625aa4d3bd8e68f0bee0072a8615ae22cfd9ef916154f58a9af85bee853334"}
Sep 08 17:00:09.451849 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:09.451822    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"
Sep 08 17:00:10.332221 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:10.332196    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g"
Sep 08 17:00:10.332376 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:10.332189    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814780-8fd9g" event={"ID":"bd8a28ca-81a5-450b-bab2-9f5041a4cb23","Type":"ContainerDied","Data":"2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb"}
Sep 08 17:00:10.332376 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:10.332333    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2baccc3ce9b506ebcb1abbe7cea87b263a0e37e1c3ba275c625d8ca0062d4beb"
Sep 08 17:00:12.936573 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:12.936530    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:12.936961 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:12.936614    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:12.949335 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:12.949290    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:13.354356 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:13.354249    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g"
Sep 08 17:00:27.510971 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.510937    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"]
Sep 08 17:00:27.511347 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.511269    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd8a28ca-81a5-450b-bab2-9f5041a4cb23" containerName="cleanup"
Sep 08 17:00:27.511347 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.511281    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd8a28ca-81a5-450b-bab2-9f5041a4cb23" containerName="cleanup"
Sep 08 17:00:27.511428 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.511354    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd8a28ca-81a5-450b-bab2-9f5041a4cb23" containerName="cleanup"
Sep 08 17:00:27.514199 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.514184    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.517314 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.517296    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 08 17:00:27.531535 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.531509    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"]
Sep 08 17:00:27.570458 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570427    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvkf\" (UniqueName: \"kubernetes.io/projected/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-kube-api-access-hxvkf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.570587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570468    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.570587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570500    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.570587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570557    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.570587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570577    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.570721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.570595    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671799 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671766    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671817    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671840    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671864    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671884    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvkf\" (UniqueName: \"kubernetes.io/projected/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-kube-api-access-hxvkf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.671970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.671909    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.672294 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.672260    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.672451 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.672270    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.672451 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.672371    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.674024 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.673998    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.674300 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.674281    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.688103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.688069    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvkf\" (UniqueName: \"kubernetes.io/projected/a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4-kube-api-access-hxvkf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-65t98\" (UID: \"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:27.823795 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:27.823696    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:28.107273 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:28.107234    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"]
Sep 08 17:00:28.404739 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:28.404714    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98" event={"ID":"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4","Type":"ContainerStarted","Data":"ded23051349298d294d11080685e84b9c42a5dd5fa270b8e2a1add2b33d6f229"}
Sep 08 17:00:29.410479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:29.410444    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98" event={"ID":"a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4","Type":"ContainerStarted","Data":"982bcea9611504fb1e9d05d3416702d8fd447682df64643cb568361b63433d32"}
Sep 08 17:00:29.433243 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:29.433191    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98" podStartSLOduration=2.185499701 podStartE2EDuration="2.433176679s" totalImagesPullingTime="247.676978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:00:29.431767597 +0000 UTC m=+572.898973786" watchObservedRunningTime="2026-09-08 17:00:29.433176679 +0000 UTC m=+572.900382881"
Sep 08 17:00:37.824687 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:37.824643    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:37.825170 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:37.824728    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:37.837456 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:37.837424    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:38.460019 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:38.459989    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-65t98"
Sep 08 17:00:56.964980 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:56.964948    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:00:56.965410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:00:56.965096    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:01:01.014993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.014958    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"]
Sep 08 17:01:01.016952 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.016936    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.019690 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.019658    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 17:01:01.029386 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.029361    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"]
Sep 08 17:01:01.161406 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161371    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzj8\" (UniqueName: \"kubernetes.io/projected/432ee6b2-5c1b-4df4-be17-81a23d82cd90-kube-api-access-nlzj8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.161584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161412    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.161584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161437    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.161584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161457    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.161584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161518    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.161584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.161560    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262188 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262152    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzj8\" (UniqueName: \"kubernetes.io/projected/432ee6b2-5c1b-4df4-be17-81a23d82cd90-kube-api-access-nlzj8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262188 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262189    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262212    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262241    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262277    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262300    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262652 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262630    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262653    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.262754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.262702    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.264436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.264418    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432ee6b2-5c1b-4df4-be17-81a23d82cd90-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.264599 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.264582    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432ee6b2-5c1b-4df4-be17-81a23d82cd90-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.271834 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.271776    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzj8\" (UniqueName: \"kubernetes.io/projected/432ee6b2-5c1b-4df4-be17-81a23d82cd90-kube-api-access-nlzj8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq\" (UID: \"432ee6b2-5c1b-4df4-be17-81a23d82cd90\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.326902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.326864    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:01.598204 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:01.598130    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"]
Sep 08 17:01:01.601516 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:01:01.601491    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432ee6b2_5c1b_4df4_be17_81a23d82cd90.slice/crio-30a25c7c7ff17c5f3bd24c4fe6b51ff5d472ebd067afafab11ef6a0839cb15ae WatchSource:0}: Error finding container 30a25c7c7ff17c5f3bd24c4fe6b51ff5d472ebd067afafab11ef6a0839cb15ae: Status 404 returned error can't find the container with id 30a25c7c7ff17c5f3bd24c4fe6b51ff5d472ebd067afafab11ef6a0839cb15ae
Sep 08 17:01:02.545202 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:02.545166    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq" event={"ID":"432ee6b2-5c1b-4df4-be17-81a23d82cd90","Type":"ContainerStarted","Data":"00a48479908ecdb14b259de1e86fc3ca5ee89045a8b8aaaeb9e1d470aa4e9e3d"}
Sep 08 17:01:02.545627 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:02.545222    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq" event={"ID":"432ee6b2-5c1b-4df4-be17-81a23d82cd90","Type":"ContainerStarted","Data":"30a25c7c7ff17c5f3bd24c4fe6b51ff5d472ebd067afafab11ef6a0839cb15ae"}
Sep 08 17:01:02.564096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:02.564032    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq" podStartSLOduration=2.374658808 podStartE2EDuration="2.564009298s" totalImagesPullingTime="189.35049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:01:02.563261953 +0000 UTC m=+606.030468147" watchObservedRunningTime="2026-09-08 17:01:02.564009298 +0000 UTC m=+606.031215483"
Sep 08 17:01:11.327271 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:11.327229    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:11.327803 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:11.327298    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:11.339803 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:11.339775    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:11.591917 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:11.591833    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq"
Sep 08 17:01:33.905907 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:33.905863    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"]
Sep 08 17:01:33.908604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:33.908583    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:33.911406 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:33.911376    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 17:01:33.926759 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:33.926732    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"]
Sep 08 17:01:34.026591 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026553    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.026591 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026598    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvg45\" (UniqueName: \"kubernetes.io/projected/3551ab1c-99b9-491a-a4b1-6735290cc656-kube-api-access-qvg45\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.026825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026625    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3551ab1c-99b9-491a-a4b1-6735290cc656-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.026825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026684    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.026825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026710    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.026825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.026752    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127489 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127449    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3551ab1c-99b9-491a-a4b1-6735290cc656-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127496    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127524    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127575    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127593    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127619    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvg45\" (UniqueName: \"kubernetes.io/projected/3551ab1c-99b9-491a-a4b1-6735290cc656-kube-api-access-qvg45\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127984 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127955    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.127984 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.127983    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.128133 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.128009    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.129842 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.129822    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3551ab1c-99b9-491a-a4b1-6735290cc656-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.129985 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.129969    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3551ab1c-99b9-491a-a4b1-6735290cc656-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.137133 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.137104    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvg45\" (UniqueName: \"kubernetes.io/projected/3551ab1c-99b9-491a-a4b1-6735290cc656-kube-api-access-qvg45\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz\" (UID: \"3551ab1c-99b9-491a-a4b1-6735290cc656\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.219675 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.219634    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:34.491739 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.491665    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"]
Sep 08 17:01:34.493970 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:01:34.493939    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3551ab1c_99b9_491a_a4b1_6735290cc656.slice/crio-05315b77e1cec8140d8536a3491b60bd92fd28391046a25c2d2883b2aa3c6a60 WatchSource:0}: Error finding container 05315b77e1cec8140d8536a3491b60bd92fd28391046a25c2d2883b2aa3c6a60: Status 404 returned error can't find the container with id 05315b77e1cec8140d8536a3491b60bd92fd28391046a25c2d2883b2aa3c6a60
Sep 08 17:01:34.495418 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.495401    2542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 17:01:34.677507 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:34.677474    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz" event={"ID":"3551ab1c-99b9-491a-a4b1-6735290cc656","Type":"ContainerStarted","Data":"05315b77e1cec8140d8536a3491b60bd92fd28391046a25c2d2883b2aa3c6a60"}
Sep 08 17:01:35.683079 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:35.683044    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz" event={"ID":"3551ab1c-99b9-491a-a4b1-6735290cc656","Type":"ContainerStarted","Data":"801adf60f05bc942f1cab2860c52f31d9a101ea0adbe9f2cc68b910f31aef3cc"}
Sep 08 17:01:35.704069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:35.704022    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz" podStartSLOduration=2.549177234 podStartE2EDuration="2.704005407s" totalImagesPullingTime="154.828173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:01:35.702912546 +0000 UTC m=+639.170118738" watchObservedRunningTime="2026-09-08 17:01:35.704005407 +0000 UTC m=+639.171211600"
Sep 08 17:01:44.219759 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:44.219722    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:44.220382 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:44.219780    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:44.232453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:44.232425    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:01:44.729501 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:01:44.729473    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz"
Sep 08 17:03:05.224700 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.224601    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:03:05.226821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.226797    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.231963 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.231942    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2746ad\""
Sep 08 17:03:05.256398 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.256364    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:03:05.295678 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.295639    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:03:05.297930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.297910    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.300682 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.300660    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2746ad-dockercfg-s2mcv\""
Sep 08 17:03:05.323344 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.323289    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqnjp\" (UniqueName: \"kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.323495 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.323456    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.323856 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.323836    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:03:05.423881 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.423838    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.423881 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.423885    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.424100 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.423958    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlvs\" (UniqueName: \"kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.424100 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.424008    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqnjp\" (UniqueName: \"kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.424542 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.424525    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.436708 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.436686    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqnjp\" (UniqueName: \"kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp\") pod \"payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.525028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.524936    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.525028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.525007    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlvs\" (UniqueName: \"kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.525690 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.525670    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.536560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.536533    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlvs\" (UniqueName: \"kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs\") pod \"payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.536653 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.536640    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:05.607850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.607817    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:05.820602 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.820574    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:03:05.820831 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:05.820804    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56281f02_3298_4744_9a0b_4d278a5e4f70.slice/crio-76c8537c420c23b1af8c97a251171d00d93beeaaf85fdb3d3b79a61cbd9d9cfc WatchSource:0}: Error finding container 76c8537c420c23b1af8c97a251171d00d93beeaaf85fdb3d3b79a61cbd9d9cfc: Status 404 returned error can't find the container with id 76c8537c420c23b1af8c97a251171d00d93beeaaf85fdb3d3b79a61cbd9d9cfc
Sep 08 17:03:05.894685 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:05.894655    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:03:05.897770 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:05.897743    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc370cf4c_0c68_471c_91c1_e3b61c9fb769.slice/crio-fbc37610f67d3e1ad0a120a7525e0898eb0be24959ab5367207f2ec3c172c019 WatchSource:0}: Error finding container fbc37610f67d3e1ad0a120a7525e0898eb0be24959ab5367207f2ec3c172c019: Status 404 returned error can't find the container with id fbc37610f67d3e1ad0a120a7525e0898eb0be24959ab5367207f2ec3c172c019
Sep 08 17:03:06.055844 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.055808    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" event={"ID":"c370cf4c-0c68-471c-91c1-e3b61c9fb769","Type":"ContainerStarted","Data":"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"}
Sep 08 17:03:06.056038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.055972    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:06.056038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.056014    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" event={"ID":"c370cf4c-0c68-471c-91c1-e3b61c9fb769","Type":"ContainerStarted","Data":"fbc37610f67d3e1ad0a120a7525e0898eb0be24959ab5367207f2ec3c172c019"}
Sep 08 17:03:06.057529 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.057497    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" event={"ID":"56281f02-3298-4744-9a0b-4d278a5e4f70","Type":"ContainerStarted","Data":"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"}
Sep 08 17:03:06.057635 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.057544    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:06.057635 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.057562    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" event={"ID":"56281f02-3298-4744-9a0b-4d278a5e4f70","Type":"ContainerStarted","Data":"76c8537c420c23b1af8c97a251171d00d93beeaaf85fdb3d3b79a61cbd9d9cfc"}
Sep 08 17:03:06.084385 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:06.084275    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" podStartSLOduration=1.084259051 podStartE2EDuration="1.084259051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:06.08291933 +0000 UTC m=+729.550125526" watchObservedRunningTime="2026-09-08 17:03:06.084259051 +0000 UTC m=+729.551465243"
Sep 08 17:03:13.529913 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.529847    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" podStartSLOduration=8.529829405 podStartE2EDuration="8.529829405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:06.12813365 +0000 UTC m=+729.595339842" watchObservedRunningTime="2026-09-08 17:03:13.529829405 +0000 UTC m=+736.997035599"
Sep 08 17:03:13.530568 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.530551    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:13.533215 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.533191    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.539476 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.539449    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a57edc2e\""
Sep 08 17:03:13.540132 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.540079    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a57edc2e-dockercfg-lq7vt\""
Sep 08 17:03:13.567599 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.567558    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:13.588474 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.588436    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:13.590748 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.590723    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.601947 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.601921    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmb78\" (UniqueName: \"kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.602064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.602016    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.639821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.639788    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:13.703070 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.703034    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.703236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.703083    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.703236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.703118    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljjq\" (UniqueName: \"kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.703236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.703146    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmb78\" (UniqueName: \"kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.703733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.703714    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.714560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.714529    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmb78\" (UniqueName: \"kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78\") pod \"payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.804635 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.804534    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljjq\" (UniqueName: \"kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.804635 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.804625    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.805287 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.805266    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.814405 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.814370    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljjq\" (UniqueName: \"kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq\") pod \"payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") " pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:13.844227 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.844188    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:13.900698 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:13.900663    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:14.123642 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:14.123563    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:14.126732 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:14.126698    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603be5ff_3aeb_4711_a251_f41d21017786.slice/crio-72edbbb46efe9698cffd13ab52a68826e95e5947d3c6e9b5bb7306384787ba72 WatchSource:0}: Error finding container 72edbbb46efe9698cffd13ab52a68826e95e5947d3c6e9b5bb7306384787ba72: Status 404 returned error can't find the container with id 72edbbb46efe9698cffd13ab52a68826e95e5947d3c6e9b5bb7306384787ba72
Sep 08 17:03:14.196604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:14.196542    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:14.198834 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:14.198797    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7e6cfc_6c3e_4349_8991_20fc81ad02ec.slice/crio-7dc27d02d8d1b1fce565bb21683189ea2aa02d4da39007587dc6b7fbc7969bed WatchSource:0}: Error finding container 7dc27d02d8d1b1fce565bb21683189ea2aa02d4da39007587dc6b7fbc7969bed: Status 404 returned error can't find the container with id 7dc27d02d8d1b1fce565bb21683189ea2aa02d4da39007587dc6b7fbc7969bed
Sep 08 17:03:15.099589 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.099553    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" event={"ID":"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec","Type":"ContainerStarted","Data":"8dfcbb2a98eb4fad45451d8401188c50a46e206e35652a6d4095e956a4f9c073"}
Sep 08 17:03:15.100057 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.099637    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" event={"ID":"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec","Type":"ContainerStarted","Data":"7dc27d02d8d1b1fce565bb21683189ea2aa02d4da39007587dc6b7fbc7969bed"}
Sep 08 17:03:15.100057 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.099687    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:15.101351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.101305    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" event={"ID":"603be5ff-3aeb-4711-a251-f41d21017786","Type":"ContainerStarted","Data":"9dfcf561489110aa1172332b0b0cb800b141698236dc4ae83fcd69b177065258"}
Sep 08 17:03:15.101474 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.101375    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" event={"ID":"603be5ff-3aeb-4711-a251-f41d21017786","Type":"ContainerStarted","Data":"72edbbb46efe9698cffd13ab52a68826e95e5947d3c6e9b5bb7306384787ba72"}
Sep 08 17:03:15.101474 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.101404    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:15.119366 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.119300    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" podStartSLOduration=2.119281928 podStartE2EDuration="2.119281928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:15.118626044 +0000 UTC m=+738.585832239" watchObservedRunningTime="2026-09-08 17:03:15.119281928 +0000 UTC m=+738.586488122"
Sep 08 17:03:15.146492 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:15.146438    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" podStartSLOduration=2.146420249 podStartE2EDuration="2.146420249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:15.14536031 +0000 UTC m=+738.612566497" watchObservedRunningTime="2026-09-08 17:03:15.146420249 +0000 UTC m=+738.613626441"
Sep 08 17:03:17.063944 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:17.063912    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:03:17.064374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:17.064296    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:03:20.739043 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.739009    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:03:20.742446 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.742430    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.746911 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.746887    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-66e7a8\""
Sep 08 17:03:20.746911 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.746906    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-66e7a8-dockercfg-ft4zt\""
Sep 08 17:03:20.767232 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.767204    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:03:20.770415 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.770396    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.780908 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.780880    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:03:20.790385 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.790358    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:03:20.869166 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.869129    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxxbw\" (UniqueName: \"kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.869337 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.869174    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.869337 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.869265    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4sz\" (UniqueName: \"kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.869337 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.869313    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.970490 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.970445    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4sz\" (UniqueName: \"kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.970882 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.970507    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.970882 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.970537    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxxbw\" (UniqueName: \"kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.970882 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.970559    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.971239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.971218    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.971279 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.971240    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:20.986575 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.986551    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4sz\" (UniqueName: \"kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz\") pod \"payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:20.988577 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:20.988556    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxxbw\" (UniqueName: \"kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw\") pod \"payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") " pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:21.052506 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:21.052414    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:21.082879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:21.082849    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:21.389924 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:21.389901    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:03:21.391667 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:21.391638    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9facd4_48a9_47c6_9177_ec4db9fd9251.slice/crio-b2c40bab74343e71ab71496a4b9dc9db37632d588947f97c12b2e36a00067025 WatchSource:0}: Error finding container b2c40bab74343e71ab71496a4b9dc9db37632d588947f97c12b2e36a00067025: Status 404 returned error can't find the container with id b2c40bab74343e71ab71496a4b9dc9db37632d588947f97c12b2e36a00067025
Sep 08 17:03:21.434853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:21.434820    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:03:21.436691 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:21.436668    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f487ed_9325_4725_b655_ee2d60bff056.slice/crio-473cb10457cab1e9222379d48d40115c289cddb0977bc0ae6d221a5ac531e8f4 WatchSource:0}: Error finding container 473cb10457cab1e9222379d48d40115c289cddb0977bc0ae6d221a5ac531e8f4: Status 404 returned error can't find the container with id 473cb10457cab1e9222379d48d40115c289cddb0977bc0ae6d221a5ac531e8f4
Sep 08 17:03:22.135716 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.135681    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" event={"ID":"fe9facd4-48a9-47c6-9177-ec4db9fd9251","Type":"ContainerStarted","Data":"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"}
Sep 08 17:03:22.136200 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.135863    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:22.136200 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.135900    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" event={"ID":"fe9facd4-48a9-47c6-9177-ec4db9fd9251","Type":"ContainerStarted","Data":"b2c40bab74343e71ab71496a4b9dc9db37632d588947f97c12b2e36a00067025"}
Sep 08 17:03:22.137236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.137211    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" event={"ID":"01f487ed-9325-4725-b655-ee2d60bff056","Type":"ContainerStarted","Data":"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"}
Sep 08 17:03:22.137369 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.137254    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" event={"ID":"01f487ed-9325-4725-b655-ee2d60bff056","Type":"ContainerStarted","Data":"473cb10457cab1e9222379d48d40115c289cddb0977bc0ae6d221a5ac531e8f4"}
Sep 08 17:03:22.137369 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.137343    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:22.160600 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.160549    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" podStartSLOduration=2.160532409 podStartE2EDuration="2.160532409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:22.160318527 +0000 UTC m=+745.627524722" watchObservedRunningTime="2026-09-08 17:03:22.160532409 +0000 UTC m=+745.627738603"
Sep 08 17:03:22.191468 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:22.191416    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" podStartSLOduration=2.191400869 podStartE2EDuration="2.191400869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:22.189908786 +0000 UTC m=+745.657114980" watchObservedRunningTime="2026-09-08 17:03:22.191400869 +0000 UTC m=+745.658607062"
Sep 08 17:03:26.107609 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:26.107572    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:26.108082 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:26.107786    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:33.144211 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:33.144177    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:03:33.144610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:33.144252    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:03:34.509153 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.509117    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:03:34.512730 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.512706    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.522760 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.522737    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5a7cab\""
Sep 08 17:03:34.549942 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.549909    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:03:34.693543 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.693505    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49w45\" (UniqueName: \"kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.693714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.693567    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.794487 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.794392    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.794651 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.794559    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49w45\" (UniqueName: \"kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.795140 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.795116    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.806779 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.806748    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49w45\" (UniqueName: \"kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45\") pod \"payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:34.822594 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:34.822563    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:35.107387 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:35.107289    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:03:35.110981 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:35.110951    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eecd8f_2c2e_45a1_aaf4_6ec89dc21a69.slice/crio-e6b0a7bdc82f5dc4551d5c9fffef5d3e2ebc95beac3082b5105d11dd22dedd18 WatchSource:0}: Error finding container e6b0a7bdc82f5dc4551d5c9fffef5d3e2ebc95beac3082b5105d11dd22dedd18: Status 404 returned error can't find the container with id e6b0a7bdc82f5dc4551d5c9fffef5d3e2ebc95beac3082b5105d11dd22dedd18
Sep 08 17:03:35.203093 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:35.203056    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" event={"ID":"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69","Type":"ContainerStarted","Data":"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"}
Sep 08 17:03:35.203264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:35.203118    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" event={"ID":"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69","Type":"ContainerStarted","Data":"e6b0a7bdc82f5dc4551d5c9fffef5d3e2ebc95beac3082b5105d11dd22dedd18"}
Sep 08 17:03:35.203264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:35.203184    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:35.237766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:35.237712    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" podStartSLOduration=1.237693409 podStartE2EDuration="1.237693409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:35.237381193 +0000 UTC m=+758.704587387" watchObservedRunningTime="2026-09-08 17:03:35.237693409 +0000 UTC m=+758.704899602"
Sep 08 17:03:39.319505 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.319473    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:03:39.323938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.323920    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.327050 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.327027    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-69b11073-kserve-self-signed-certs\""
Sep 08 17:03:39.338734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.338711    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:03:39.429711 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429666    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8vk\" (UniqueName: \"kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.429711 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429716    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.429970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429768    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.429970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429800    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.429970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429823    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.429970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.429846    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530757 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530715    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530763    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530788    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530816    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530912    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8vk\" (UniqueName: \"kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.530953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.530944    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.531233 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.531203    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.531292 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.531238    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.531369 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.531345    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.533080 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.533057    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.534045 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.534026    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.547716 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.547669    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8vk\" (UniqueName: \"kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk\") pod \"e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") " pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.632958 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.632866    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:39.919103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:39.919078    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:03:39.920408 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:03:39.920380    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe590c0c_d293_40f4_a76a_e5e6ad70b4fe.slice/crio-7d8c27d7c102721be55c89fa8dd4af494a0bce38f49e9f006b9db4cd16507b3f WatchSource:0}: Error finding container 7d8c27d7c102721be55c89fa8dd4af494a0bce38f49e9f006b9db4cd16507b3f: Status 404 returned error can't find the container with id 7d8c27d7c102721be55c89fa8dd4af494a0bce38f49e9f006b9db4cd16507b3f
Sep 08 17:03:40.228204 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:40.228167    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" event={"ID":"be590c0c-d293-40f4-a76a-e5e6ad70b4fe","Type":"ContainerStarted","Data":"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"}
Sep 08 17:03:40.228204 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:40.228206    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" event={"ID":"be590c0c-d293-40f4-a76a-e5e6ad70b4fe","Type":"ContainerStarted","Data":"7d8c27d7c102721be55c89fa8dd4af494a0bce38f49e9f006b9db4cd16507b3f"}
Sep 08 17:03:40.464851 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:40.464797    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" podStartSLOduration=1.4647808420000001 podStartE2EDuration="1.464780842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:03:40.463816631 +0000 UTC m=+763.931022824" watchObservedRunningTime="2026-09-08 17:03:40.464780842 +0000 UTC m=+763.931987035"
Sep 08 17:03:46.210310 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:46.210278    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:03:49.633281 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:49.633242    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:49.633281 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:49.633289    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:49.646113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:49.646081    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:50.286917 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:50.286888    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:03:54.160072 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.160040    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:54.160514 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.160255    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" podUID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" containerName="payload-processing" containerID="cri-o://8dfcbb2a98eb4fad45451d8401188c50a46e206e35652a6d4095e956a4f9c073" gracePeriod=30
Sep 08 17:03:54.167769 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.167731    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:54.168005 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.167981    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" podUID="603be5ff-3aeb-4711-a251-f41d21017786" containerName="payload-pre-processing" containerID="cri-o://9dfcf561489110aa1172332b0b0cb800b141698236dc4ae83fcd69b177065258" gracePeriod=30
Sep 08 17:03:54.296573 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.296541    2542 generic.go:356] "Generic (PLEG): container finished" podID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" containerID="8dfcbb2a98eb4fad45451d8401188c50a46e206e35652a6d4095e956a4f9c073" exitCode=0
Sep 08 17:03:54.296731 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.296633    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" event={"ID":"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec","Type":"ContainerDied","Data":"8dfcbb2a98eb4fad45451d8401188c50a46e206e35652a6d4095e956a4f9c073"}
Sep 08 17:03:54.298631 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.298602    2542 generic.go:356] "Generic (PLEG): container finished" podID="603be5ff-3aeb-4711-a251-f41d21017786" containerID="9dfcf561489110aa1172332b0b0cb800b141698236dc4ae83fcd69b177065258" exitCode=0
Sep 08 17:03:54.298751 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.298678    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" event={"ID":"603be5ff-3aeb-4711-a251-f41d21017786","Type":"ContainerDied","Data":"9dfcf561489110aa1172332b0b0cb800b141698236dc4ae83fcd69b177065258"}
Sep 08 17:03:54.444118 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.444088    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:54.446560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.446533    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:54.547511 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547481    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmb78\" (UniqueName: \"kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78\") pod \"603be5ff-3aeb-4711-a251-f41d21017786\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") "
Sep 08 17:03:54.547511 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547516    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume\") pod \"603be5ff-3aeb-4711-a251-f41d21017786\" (UID: \"603be5ff-3aeb-4711-a251-f41d21017786\") "
Sep 08 17:03:54.547734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547577    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume\") pod \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") "
Sep 08 17:03:54.547734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547623    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tljjq\" (UniqueName: \"kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq\") pod \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\" (UID: \"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec\") "
Sep 08 17:03:54.547998 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547960    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" (UID: "ec7e6cfc-6c3e-4349-8991-20fc81ad02ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:03:54.547998 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.547959    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "603be5ff-3aeb-4711-a251-f41d21017786" (UID: "603be5ff-3aeb-4711-a251-f41d21017786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:03:54.549946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.549924    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78" (OuterVolumeSpecName: "kube-api-access-rmb78") pod "603be5ff-3aeb-4711-a251-f41d21017786" (UID: "603be5ff-3aeb-4711-a251-f41d21017786"). InnerVolumeSpecName "kube-api-access-rmb78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:03:54.549946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.549933    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq" (OuterVolumeSpecName: "kube-api-access-tljjq") pod "ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" (UID: "ec7e6cfc-6c3e-4349-8991-20fc81ad02ec"). InnerVolumeSpecName "kube-api-access-tljjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:03:54.649236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.649196    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:03:54.649236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.649227    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljjq\" (UniqueName: \"kubernetes.io/projected/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec-kube-api-access-tljjq\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:03:54.649236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.649237    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmb78\" (UniqueName: \"kubernetes.io/projected/603be5ff-3aeb-4711-a251-f41d21017786-kube-api-access-rmb78\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:03:54.649236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:54.649246    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/603be5ff-3aeb-4711-a251-f41d21017786-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:03:55.304498 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.304464    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq" event={"ID":"ec7e6cfc-6c3e-4349-8991-20fc81ad02ec","Type":"ContainerDied","Data":"7dc27d02d8d1b1fce565bb21683189ea2aa02d4da39007587dc6b7fbc7969bed"}
Sep 08 17:03:55.304923 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.304476    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"
Sep 08 17:03:55.304923 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.304528    2542 scope.go:117] "RemoveContainer" containerID="8dfcbb2a98eb4fad45451d8401188c50a46e206e35652a6d4095e956a4f9c073"
Sep 08 17:03:55.305921 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.305898    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"
Sep 08 17:03:55.306016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.305896    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv" event={"ID":"603be5ff-3aeb-4711-a251-f41d21017786","Type":"ContainerDied","Data":"72edbbb46efe9698cffd13ab52a68826e95e5947d3c6e9b5bb7306384787ba72"}
Sep 08 17:03:55.322376 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.322011    2542 scope.go:117] "RemoveContainer" containerID="9dfcf561489110aa1172332b0b0cb800b141698236dc4ae83fcd69b177065258"
Sep 08 17:03:55.382846 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.382771    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:55.397111 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.397080    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a57edc2e-6884bb9954-j8cxv"]
Sep 08 17:03:55.446754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.446720    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:55.460005 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:55.459979    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a57edc2e-6b9b75f48c-zjlmq"]
Sep 08 17:03:57.115787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:57.115749    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603be5ff-3aeb-4711-a251-f41d21017786" path="/var/lib/kubelet/pods/603be5ff-3aeb-4711-a251-f41d21017786/volumes"
Sep 08 17:03:57.116122 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:03:57.116097    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" path="/var/lib/kubelet/pods/ec7e6cfc-6c3e-4349-8991-20fc81ad02ec/volumes"
Sep 08 17:04:31.123471 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123440    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123794    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603be5ff-3aeb-4711-a251-f41d21017786" containerName="payload-pre-processing"
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123808    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="603be5ff-3aeb-4711-a251-f41d21017786" containerName="payload-pre-processing"
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123824    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" containerName="payload-processing"
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123830    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" containerName="payload-processing"
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123880    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="603be5ff-3aeb-4711-a251-f41d21017786" containerName="payload-pre-processing"
Sep 08 17:04:31.123922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.123892    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec7e6cfc-6c3e-4349-8991-20fc81ad02ec" containerName="payload-processing"
Sep 08 17:04:31.126903 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.126887    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.130528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.130509    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-69b11073-kserve-self-signed-certs\""
Sep 08 17:04:31.152940 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.152914    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:04:31.283791 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283754    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.283993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283810    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.283993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283859    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.283993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283886    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.283993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283911    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mkm\" (UniqueName: \"kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.283993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.283963    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384505 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384411    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384505 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384449    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mkm\" (UniqueName: \"kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384700 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384632    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384756 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384712    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384808 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384757    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384868 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384818    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.384868 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384815    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.385012 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.384992    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.385076 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.385059    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.386901 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.386878    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.387251 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.387229    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.396904 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.396880    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mkm\" (UniqueName: \"kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm\") pod \"e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") " pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.435964 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.435928    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:31.727690 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:31.727656    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:04:31.730719 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:04:31.730692    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a404aa_1491_4a23_8a44_7a9521ba026f.slice/crio-e8c72d56b89a2946d69de8d92e611c641e1c257fb654600ef209cbd5cb53ff30 WatchSource:0}: Error finding container e8c72d56b89a2946d69de8d92e611c641e1c257fb654600ef209cbd5cb53ff30: Status 404 returned error can't find the container with id e8c72d56b89a2946d69de8d92e611c641e1c257fb654600ef209cbd5cb53ff30
Sep 08 17:04:32.474679 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:32.474646    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" event={"ID":"28a404aa-1491-4a23-8a44-7a9521ba026f","Type":"ContainerStarted","Data":"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"}
Sep 08 17:04:32.474679 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:32.474687    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" event={"ID":"28a404aa-1491-4a23-8a44-7a9521ba026f","Type":"ContainerStarted","Data":"e8c72d56b89a2946d69de8d92e611c641e1c257fb654600ef209cbd5cb53ff30"}
Sep 08 17:04:32.501909 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:32.501856    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" podStartSLOduration=1.501841005 podStartE2EDuration="1.501841005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:04:32.499711829 +0000 UTC m=+815.966918024" watchObservedRunningTime="2026-09-08 17:04:32.501841005 +0000 UTC m=+815.969047250"
Sep 08 17:04:41.436865 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:41.436820    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:41.437352 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:41.436885    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:41.449747 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:41.449716    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:04:41.527388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:04:41.527358    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:05:01.485955 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.485916    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"]
Sep 08 17:05:01.489175 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.489156    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:01.494162 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.494141    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:05:01.494162 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.494149    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:05:01.513364 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.513319    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"]
Sep 08 17:05:01.674409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.674377    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f155bce9-ms684\" (UID: \"9fc259ed-6730-41ba-986a-f18f31aa0312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:01.775413 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.775309    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f155bce9-ms684\" (UID: \"9fc259ed-6730-41ba-986a-f18f31aa0312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:01.775917 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.775895    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f155bce9-ms684\" (UID: \"9fc259ed-6730-41ba-986a-f18f31aa0312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:01.798279 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:01.798252    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:02.082236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.082149    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"]
Sep 08 17:05:02.087687 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:05:02.087655    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc259ed_6730_41ba_986a_f18f31aa0312.slice/crio-c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe WatchSource:0}: Error finding container c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe: Status 404 returned error can't find the container with id c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe
Sep 08 17:05:02.324362 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.324319    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:02.328675 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.328654    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.332191 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.332169    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-66e7a8\"/\"kube-root-ca.crt\""
Sep 08 17:05:02.333193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.333132    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-66e7a8\"/\"autores-model-66e7a8-kserve-self-signed-certs\""
Sep 08 17:05:02.333193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.333163    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-66e7a8\"/\"default-dockercfg-4pp59\""
Sep 08 17:05:02.333193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.333170    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-66e7a8\"/\"openshift-service-ca.crt\""
Sep 08 17:05:02.345108 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.345088    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:02.482290 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482255    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.482290 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482291    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.482513 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482318    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.482513 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482395    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz54t\" (UniqueName: \"kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.482513 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482490    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.482605 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.482530    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.583974 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.583886    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.583974 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.583929    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.583974 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.583957    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584067    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584116    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584168    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz54t\" (UniqueName: \"kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584281    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584407    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.584522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.584510    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.586849 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.586823    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.586991 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.586974    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.593606 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.593590    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz54t\" (UniqueName: \"kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t\") pod \"autores-model-66e7a8-kserve-547768fb55-55vbq\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") " pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.612113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.612079    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684" event={"ID":"9fc259ed-6730-41ba-986a-f18f31aa0312","Type":"ContainerStarted","Data":"2f65976cee4c3a613c304515d3ca13b859ced68b42350d1dbae7c90001574672"}
Sep 08 17:05:02.612223 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.612153    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684" event={"ID":"9fc259ed-6730-41ba-986a-f18f31aa0312","Type":"ContainerStarted","Data":"c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe"}
Sep 08 17:05:02.637369 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.637297    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:02.638290 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.638252    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684" podStartSLOduration=1.638238954 podStartE2EDuration="1.638238954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:02.636131363 +0000 UTC m=+846.103337555" watchObservedRunningTime="2026-09-08 17:05:02.638238954 +0000 UTC m=+846.105445147"
Sep 08 17:05:02.923106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:02.922803    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:02.925703 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:05:02.925676    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a675188_483b_4452_b0bb_e3456f958300.slice/crio-a65233e441f25dde16de39dcaafb1fbc1a63b945870eda7184f004e59e631d32 WatchSource:0}: Error finding container a65233e441f25dde16de39dcaafb1fbc1a63b945870eda7184f004e59e631d32: Status 404 returned error can't find the container with id a65233e441f25dde16de39dcaafb1fbc1a63b945870eda7184f004e59e631d32
Sep 08 17:05:03.617946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:03.617906    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" event={"ID":"4a675188-483b-4452-b0bb-e3456f958300","Type":"ContainerStarted","Data":"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"}
Sep 08 17:05:03.618421 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:03.617962    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" event={"ID":"4a675188-483b-4452-b0bb-e3456f958300","Type":"ContainerStarted","Data":"a65233e441f25dde16de39dcaafb1fbc1a63b945870eda7184f004e59e631d32"}
Sep 08 17:05:03.619362 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:03.619341    2542 generic.go:356] "Generic (PLEG): container finished" podID="9fc259ed-6730-41ba-986a-f18f31aa0312" containerID="2f65976cee4c3a613c304515d3ca13b859ced68b42350d1dbae7c90001574672" exitCode=0
Sep 08 17:05:03.619452 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:03.619371    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684" event={"ID":"9fc259ed-6730-41ba-986a-f18f31aa0312","Type":"ContainerDied","Data":"2f65976cee4c3a613c304515d3ca13b859ced68b42350d1dbae7c90001574672"}
Sep 08 17:05:03.655747 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:03.655700    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" podStartSLOduration=1.655682427 podStartE2EDuration="1.655682427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:03.655468356 +0000 UTC m=+847.122674550" watchObservedRunningTime="2026-09-08 17:05:03.655682427 +0000 UTC m=+847.122888619"
Sep 08 17:05:04.748724 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:04.748700    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:04.912460 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:04.912375    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca\") pod \"9fc259ed-6730-41ba-986a-f18f31aa0312\" (UID: \"9fc259ed-6730-41ba-986a-f18f31aa0312\") "
Sep 08 17:05:04.912731 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:04.912708    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9fc259ed-6730-41ba-986a-f18f31aa0312" (UID: "9fc259ed-6730-41ba-986a-f18f31aa0312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:05:05.013370 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:05.013309    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fc259ed-6730-41ba-986a-f18f31aa0312-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:05.629389 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:05.629351    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684" event={"ID":"9fc259ed-6730-41ba-986a-f18f31aa0312","Type":"ContainerDied","Data":"c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe"}
Sep 08 17:05:05.629557 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:05.629409    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2cf116a1208b863d1379c5ad1775ddef9853a38880a179d6b83e722eb06ffbe"
Sep 08 17:05:05.629557 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:05.629364    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"
Sep 08 17:05:07.976009 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:07.975938    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"]
Sep 08 17:05:07.978061 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:07.978039    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f155bce9-ms684"]
Sep 08 17:05:09.117576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:09.117539    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fc259ed-6730-41ba-986a-f18f31aa0312" path="/var/lib/kubelet/pods/9fc259ed-6730-41ba-986a-f18f31aa0312/volumes"
Sep 08 17:05:12.637446 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:12.637408    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:12.637834 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:12.637458    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:12.650122 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:12.650096    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:12.674606 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:12.674576    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:32.135073 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.135041    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:05:32.135541 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.135472    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fc259ed-6730-41ba-986a-f18f31aa0312" containerName="revoke-keys"
Sep 08 17:05:32.135541 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.135485    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fc259ed-6730-41ba-986a-f18f31aa0312" containerName="revoke-keys"
Sep 08 17:05:32.135620 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.135553    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fc259ed-6730-41ba-986a-f18f31aa0312" containerName="revoke-keys"
Sep 08 17:05:32.138561 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.138542    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.142431 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.142407    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2746ad\"/\"openshift-service-ca.crt\""
Sep 08 17:05:32.143504 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.143481    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2746ad\"/\"default-dockercfg-bcxdf\""
Sep 08 17:05:32.143504 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.143495    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2746ad\"/\"kube-root-ca.crt\""
Sep 08 17:05:32.143644 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.143613    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2746ad\"/\"test-model-2746ad-kserve-self-signed-certs\""
Sep 08 17:05:32.160417 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.160389    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:05:32.265926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.265896    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.266077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.265931    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqzp\" (UniqueName: \"kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.266077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.266003    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.266077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.266059    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.266182 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.266082    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.266182 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.266167    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366721    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366757    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqzp\" (UniqueName: \"kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366790    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366831    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366849    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.366976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.366872    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.367273 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.367242    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.367442 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.367287    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.367442 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.367349    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.369059 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.369033    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.369210 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.369193    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.375316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.375294    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqzp\" (UniqueName: \"kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp\") pod \"test-model-2746ad-kserve-6998d755d9-xd7k5\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") " pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.448166 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.448137    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:32.722633 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.722608    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:05:32.725036 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:05:32.725009    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8219fc56_185f_4b3d_bf24_8a53a39a4e58.slice/crio-f3f51dd37931a92f2d138b4c7ae76b12ef38c42496e361f3008eea57efdebe3c WatchSource:0}: Error finding container f3f51dd37931a92f2d138b4c7ae76b12ef38c42496e361f3008eea57efdebe3c: Status 404 returned error can't find the container with id f3f51dd37931a92f2d138b4c7ae76b12ef38c42496e361f3008eea57efdebe3c
Sep 08 17:05:32.756875 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:32.756840    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" event={"ID":"8219fc56-185f-4b3d-bf24-8a53a39a4e58","Type":"ContainerStarted","Data":"f3f51dd37931a92f2d138b4c7ae76b12ef38c42496e361f3008eea57efdebe3c"}
Sep 08 17:05:33.764552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:33.764513    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" event={"ID":"8219fc56-185f-4b3d-bf24-8a53a39a4e58","Type":"ContainerStarted","Data":"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"}
Sep 08 17:05:33.788435 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:33.788387    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" podStartSLOduration=1.7883700500000002 podStartE2EDuration="1.78837005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:33.786424578 +0000 UTC m=+877.253630773" watchObservedRunningTime="2026-09-08 17:05:33.78837005 +0000 UTC m=+877.255576243"
Sep 08 17:05:34.298646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:34.298610    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:05:35.482697 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:35.482663    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:05:36.572247 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.572211    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:36.803377 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.803342    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"]
Sep 08 17:05:36.806688 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.806672    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:36.809790 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.809762    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:05:36.809914 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.809886    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:05:36.822365 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.822272    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"]
Sep 08 17:05:36.908137 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:36.908100    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b709328f-5whz6\" (UID: \"82c97a36-e10a-4a27-98f7-aaa420025aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:37.009641 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.009592    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b709328f-5whz6\" (UID: \"82c97a36-e10a-4a27-98f7-aaa420025aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:37.010233 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.010212    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b709328f-5whz6\" (UID: \"82c97a36-e10a-4a27-98f7-aaa420025aa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:37.123391 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.123291    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:37.394656 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.394569    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"]
Sep 08 17:05:37.396926 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:05:37.396896    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c97a36_e10a_4a27_98f7_aaa420025aa9.slice/crio-0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36 WatchSource:0}: Error finding container 0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36: Status 404 returned error can't find the container with id 0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36
Sep 08 17:05:37.783755 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.783710    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6" event={"ID":"82c97a36-e10a-4a27-98f7-aaa420025aa9","Type":"ContainerStarted","Data":"f8c482f8687dad583d3713314906a15420e6be43cc5593680ae275e92054623d"}
Sep 08 17:05:37.784249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.783771    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6" event={"ID":"82c97a36-e10a-4a27-98f7-aaa420025aa9","Type":"ContainerStarted","Data":"0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36"}
Sep 08 17:05:37.800342 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:37.800277    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6" podStartSLOduration=1.800262328 podStartE2EDuration="1.800262328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:05:37.799184738 +0000 UTC m=+881.266390944" watchObservedRunningTime="2026-09-08 17:05:37.800262328 +0000 UTC m=+881.267468521"
Sep 08 17:05:38.789392 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:38.789355    2542 generic.go:356] "Generic (PLEG): container finished" podID="82c97a36-e10a-4a27-98f7-aaa420025aa9" containerID="f8c482f8687dad583d3713314906a15420e6be43cc5593680ae275e92054623d" exitCode=0
Sep 08 17:05:38.789801 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:38.789438    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6" event={"ID":"82c97a36-e10a-4a27-98f7-aaa420025aa9","Type":"ContainerDied","Data":"f8c482f8687dad583d3713314906a15420e6be43cc5593680ae275e92054623d"}
Sep 08 17:05:39.926578 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:39.926552    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:40.038235 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.038199    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca\") pod \"82c97a36-e10a-4a27-98f7-aaa420025aa9\" (UID: \"82c97a36-e10a-4a27-98f7-aaa420025aa9\") "
Sep 08 17:05:40.038569 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.038547    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82c97a36-e10a-4a27-98f7-aaa420025aa9" (UID: "82c97a36-e10a-4a27-98f7-aaa420025aa9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:05:40.139261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.139177    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82c97a36-e10a-4a27-98f7-aaa420025aa9-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:40.800404 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.800376    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"
Sep 08 17:05:40.800576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.800371    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6" event={"ID":"82c97a36-e10a-4a27-98f7-aaa420025aa9","Type":"ContainerDied","Data":"0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36"}
Sep 08 17:05:40.800576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:40.800510    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0341a6bb984cb81732786251c39186b96162aee9c47f38c6f94872375f9d7a36"
Sep 08 17:05:42.448606 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:42.448563    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:42.449041 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:42.448630    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:42.461358 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:42.461311    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:42.821172 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:42.821098    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:05:44.647064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:44.647032    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"]
Sep 08 17:05:44.649670 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:44.649636    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b709328f-5whz6"]
Sep 08 17:05:45.116183 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:45.116151    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c97a36-e10a-4a27-98f7-aaa420025aa9" path="/var/lib/kubelet/pods/82c97a36-e10a-4a27-98f7-aaa420025aa9/volumes"
Sep 08 17:05:47.191173 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.191140    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:05:47.191698 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.191396    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" podUID="01f487ed-9325-4725-b655-ee2d60bff056" containerName="payload-processing" containerID="cri-o://ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3" gracePeriod=30
Sep 08 17:05:47.211312 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.211282    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:05:47.211837 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.211801    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" podUID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" containerName="payload-pre-processing" containerID="cri-o://f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72" gracePeriod=30
Sep 08 17:05:47.471587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.471564    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:05:47.489121 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.489097    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:05:47.508126 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508102    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm4sz\" (UniqueName: \"kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz\") pod \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") "
Sep 08 17:05:47.508249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508155    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume\") pod \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\" (UID: \"fe9facd4-48a9-47c6-9177-ec4db9fd9251\") "
Sep 08 17:05:47.508249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508173    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxxbw\" (UniqueName: \"kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw\") pod \"01f487ed-9325-4725-b655-ee2d60bff056\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") "
Sep 08 17:05:47.508249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508199    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume\") pod \"01f487ed-9325-4725-b655-ee2d60bff056\" (UID: \"01f487ed-9325-4725-b655-ee2d60bff056\") "
Sep 08 17:05:47.508606 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508578    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe9facd4-48a9-47c6-9177-ec4db9fd9251" (UID: "fe9facd4-48a9-47c6-9177-ec4db9fd9251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:05:47.508721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.508688    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01f487ed-9325-4725-b655-ee2d60bff056" (UID: "01f487ed-9325-4725-b655-ee2d60bff056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:05:47.510134 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.510111    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz" (OuterVolumeSpecName: "kube-api-access-mm4sz") pod "fe9facd4-48a9-47c6-9177-ec4db9fd9251" (UID: "fe9facd4-48a9-47c6-9177-ec4db9fd9251"). InnerVolumeSpecName "kube-api-access-mm4sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:47.510308 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.510288    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw" (OuterVolumeSpecName: "kube-api-access-jxxbw") pod "01f487ed-9325-4725-b655-ee2d60bff056" (UID: "01f487ed-9325-4725-b655-ee2d60bff056"). InnerVolumeSpecName "kube-api-access-jxxbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:47.609205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.609169    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe9facd4-48a9-47c6-9177-ec4db9fd9251-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:47.609205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.609202    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxxbw\" (UniqueName: \"kubernetes.io/projected/01f487ed-9325-4725-b655-ee2d60bff056-kube-api-access-jxxbw\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:47.609205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.609213    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f487ed-9325-4725-b655-ee2d60bff056-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:47.609205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.609221    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4sz\" (UniqueName: \"kubernetes.io/projected/fe9facd4-48a9-47c6-9177-ec4db9fd9251-kube-api-access-mm4sz\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:47.837026 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.836942    2542 generic.go:356] "Generic (PLEG): container finished" podID="01f487ed-9325-4725-b655-ee2d60bff056" containerID="ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3" exitCode=0
Sep 08 17:05:47.837026 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.836998    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"
Sep 08 17:05:47.837374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.837028    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" event={"ID":"01f487ed-9325-4725-b655-ee2d60bff056","Type":"ContainerDied","Data":"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"}
Sep 08 17:05:47.837374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.837085    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs" event={"ID":"01f487ed-9325-4725-b655-ee2d60bff056","Type":"ContainerDied","Data":"473cb10457cab1e9222379d48d40115c289cddb0977bc0ae6d221a5ac531e8f4"}
Sep 08 17:05:47.837374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.837117    2542 scope.go:117] "RemoveContainer" containerID="ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"
Sep 08 17:05:47.838529 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.838506    2542 generic.go:356] "Generic (PLEG): container finished" podID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" containerID="f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72" exitCode=0
Sep 08 17:05:47.838646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.838537    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" event={"ID":"fe9facd4-48a9-47c6-9177-ec4db9fd9251","Type":"ContainerDied","Data":"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"}
Sep 08 17:05:47.838646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.838550    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"
Sep 08 17:05:47.838646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.838593    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg" event={"ID":"fe9facd4-48a9-47c6-9177-ec4db9fd9251","Type":"ContainerDied","Data":"b2c40bab74343e71ab71496a4b9dc9db37632d588947f97c12b2e36a00067025"}
Sep 08 17:05:47.847678 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.847657    2542 scope.go:117] "RemoveContainer" containerID="ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"
Sep 08 17:05:47.847999 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:05:47.847979    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3\": container with ID starting with ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3 not found: ID does not exist" containerID="ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"
Sep 08 17:05:47.848045 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.848007    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3"} err="failed to get container status \"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3\": rpc error: code = NotFound desc = could not find container \"ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3\": container with ID starting with ecd58473f980b9660d1ad8deee8499a7df908b956e8d0e6b45d84e91e67b4ef3 not found: ID does not exist"
Sep 08 17:05:47.848045 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.848035    2542 scope.go:117] "RemoveContainer" containerID="f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"
Sep 08 17:05:47.855512 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.855497    2542 scope.go:117] "RemoveContainer" containerID="f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"
Sep 08 17:05:47.855744 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:05:47.855726    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72\": container with ID starting with f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72 not found: ID does not exist" containerID="f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"
Sep 08 17:05:47.855809 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.855750    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72"} err="failed to get container status \"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72\": rpc error: code = NotFound desc = could not find container \"f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72\": container with ID starting with f3ab39dfa2d17729fd1ff4ee9ae5de70e9da1a18ef172b0555407268d122ee72 not found: ID does not exist"
Sep 08 17:05:47.876013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.875990    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:05:47.884004 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.883986    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-66e7a8-68f6b9b6db-rxfzg"]
Sep 08 17:05:47.902711 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.902687    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:05:47.911872 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:47.911850    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-66e7a8-7cbc467cb7-rl9gs"]
Sep 08 17:05:49.116687 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.116598    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f487ed-9325-4725-b655-ee2d60bff056" path="/var/lib/kubelet/pods/01f487ed-9325-4725-b655-ee2d60bff056/volumes"
Sep 08 17:05:49.117139 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.117098    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" path="/var/lib/kubelet/pods/fe9facd4-48a9-47c6-9177-ec4db9fd9251/volumes"
Sep 08 17:05:49.317763 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.317720    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" podUID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" containerName="main" containerID="cri-o://e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb" gracePeriod=45
Sep 08 17:05:49.591538 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.591518    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:05:49.623854 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623824    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623870    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623929    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8vk\" (UniqueName: \"kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623951    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623966    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.623986    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm\") pod \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\" (UID: \"be590c0c-d293-40f4-a76a-e5e6ad70b4fe\") "
Sep 08 17:05:49.624248 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.624107    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:49.624248 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.624151    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home" (OuterVolumeSpecName: "home") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:49.624248 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.624243    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.624405 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.624262    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.624405 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.624267    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:49.626314 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.626285    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:05:49.626314 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.626289    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm" (OuterVolumeSpecName: "dshm") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:49.626469 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.626364    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk" (OuterVolumeSpecName: "kube-api-access-xp8vk") pod "be590c0c-d293-40f4-a76a-e5e6ad70b4fe" (UID: "be590c0c-d293-40f4-a76a-e5e6ad70b4fe"). InnerVolumeSpecName "kube-api-access-xp8vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:49.724886 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.724790    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8vk\" (UniqueName: \"kubernetes.io/projected/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-kube-api-access-xp8vk\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.724886 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.724823    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.724886 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.724838    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.724886 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.724849    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be590c0c-d293-40f4-a76a-e5e6ad70b4fe-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:49.850660 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.850623    2542 generic.go:356] "Generic (PLEG): container finished" podID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" containerID="e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb" exitCode=0
Sep 08 17:05:49.850660 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.850664    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"
Sep 08 17:05:49.850879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.850706    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" event={"ID":"be590c0c-d293-40f4-a76a-e5e6ad70b4fe","Type":"ContainerDied","Data":"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"}
Sep 08 17:05:49.850879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.850744    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx" event={"ID":"be590c0c-d293-40f4-a76a-e5e6ad70b4fe","Type":"ContainerDied","Data":"7d8c27d7c102721be55c89fa8dd4af494a0bce38f49e9f006b9db4cd16507b3f"}
Sep 08 17:05:49.850879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.850763    2542 scope.go:117] "RemoveContainer" containerID="e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"
Sep 08 17:05:49.865052 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.865034    2542 scope.go:117] "RemoveContainer" containerID="e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"
Sep 08 17:05:49.865358 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:05:49.865336    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb\": container with ID starting with e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb not found: ID does not exist" containerID="e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"
Sep 08 17:05:49.865407 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.865367    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb"} err="failed to get container status \"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb\": rpc error: code = NotFound desc = could not find container \"e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb\": container with ID starting with e19455f8d91d56f0b640f37169a7997688a8231eb10f1a216451bc864dec7acb not found: ID does not exist"
Sep 08 17:05:49.878191 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.878170    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:05:49.882128 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:49.882097    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-69b11073-kserve-6c665897b4-th2rx"]
Sep 08 17:05:50.501792 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.501746    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" podUID="28a404aa-1491-4a23-8a44-7a9521ba026f" containerName="main" containerID="cri-o://d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78" gracePeriod=45
Sep 08 17:05:50.740174 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.740150    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:05:50.831279 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831195    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831279 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831255    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831293    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831342    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831403    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6mkm\" (UniqueName: \"kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831428    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir\") pod \"28a404aa-1491-4a23-8a44-7a9521ba026f\" (UID: \"28a404aa-1491-4a23-8a44-7a9521ba026f\") "
Sep 08 17:05:50.831721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831563    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home" (OuterVolumeSpecName: "home") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:50.831721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831583    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache" (OuterVolumeSpecName: "model-cache") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:50.831816 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831742    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:50.831816 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831759    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:50.831816 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.831778    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:50.833454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.833428    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm" (OuterVolumeSpecName: "dshm") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:50.833540 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.833436    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:05:50.833540 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.833520    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm" (OuterVolumeSpecName: "kube-api-access-t6mkm") pod "28a404aa-1491-4a23-8a44-7a9521ba026f" (UID: "28a404aa-1491-4a23-8a44-7a9521ba026f"). InnerVolumeSpecName "kube-api-access-t6mkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:50.857673 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.857630    2542 generic.go:356] "Generic (PLEG): container finished" podID="28a404aa-1491-4a23-8a44-7a9521ba026f" containerID="d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78" exitCode=0
Sep 08 17:05:50.857853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.857687    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"
Sep 08 17:05:50.857853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.857705    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" event={"ID":"28a404aa-1491-4a23-8a44-7a9521ba026f","Type":"ContainerDied","Data":"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"}
Sep 08 17:05:50.857853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.857744    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj" event={"ID":"28a404aa-1491-4a23-8a44-7a9521ba026f","Type":"ContainerDied","Data":"e8c72d56b89a2946d69de8d92e611c641e1c257fb654600ef209cbd5cb53ff30"}
Sep 08 17:05:50.857853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.857761    2542 scope.go:117] "RemoveContainer" containerID="d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"
Sep 08 17:05:50.866646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.866629    2542 scope.go:117] "RemoveContainer" containerID="d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"
Sep 08 17:05:50.866907 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:05:50.866890    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78\": container with ID starting with d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78 not found: ID does not exist" containerID="d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"
Sep 08 17:05:50.866959 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.866914    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78"} err="failed to get container status \"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78\": rpc error: code = NotFound desc = could not find container \"d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78\": container with ID starting with d4461e3e6410e35ba6ee14a8ea62361c762cab5cd202b278264a0f1b8fbe8e78 not found: ID does not exist"
Sep 08 17:05:50.879935 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.879909    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:05:50.891175 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.891147    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-69b11073-kserve-f9d857d66-rrbgj"]
Sep 08 17:05:50.932558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.932530    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:50.932558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.932556    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mkm\" (UniqueName: \"kubernetes.io/projected/28a404aa-1491-4a23-8a44-7a9521ba026f-kube-api-access-t6mkm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:50.932762 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.932566    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a404aa-1491-4a23-8a44-7a9521ba026f-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:50.932762 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:50.932579    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a404aa-1491-4a23-8a44-7a9521ba026f-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.116803 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.116722    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a404aa-1491-4a23-8a44-7a9521ba026f" path="/var/lib/kubelet/pods/28a404aa-1491-4a23-8a44-7a9521ba026f/volumes"
Sep 08 17:05:51.117118 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.117105    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" path="/var/lib/kubelet/pods/be590c0c-d293-40f4-a76a-e5e6ad70b4fe/volumes"
Sep 08 17:05:51.592821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.592783    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" podUID="4a675188-483b-4452-b0bb-e3456f958300" containerName="main" containerID="cri-o://091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3" gracePeriod=45
Sep 08 17:05:51.835270 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.835246    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:51.838975 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.838953    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839036 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.838999    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839081 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839037    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839081 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839073    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839165 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839101    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz54t\" (UniqueName: \"kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839162    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir\") pod \"4a675188-483b-4452-b0bb-e3456f958300\" (UID: \"4a675188-483b-4452-b0bb-e3456f958300\") "
Sep 08 17:05:51.839366 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839340    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:51.839455 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839358    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home" (OuterVolumeSpecName: "home") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:51.839519 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839474    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.839519 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839488    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.839519 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.839486    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:51.841413 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.841381    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm" (OuterVolumeSpecName: "dshm") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:05:51.841413 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.841387    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t" (OuterVolumeSpecName: "kube-api-access-mz54t") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "kube-api-access-mz54t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:05:51.841576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.841415    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a675188-483b-4452-b0bb-e3456f958300" (UID: "4a675188-483b-4452-b0bb-e3456f958300"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:05:51.864783 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.864753    2542 generic.go:356] "Generic (PLEG): container finished" podID="4a675188-483b-4452-b0bb-e3456f958300" containerID="091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3" exitCode=0
Sep 08 17:05:51.864938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.864793    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" event={"ID":"4a675188-483b-4452-b0bb-e3456f958300","Type":"ContainerDied","Data":"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"}
Sep 08 17:05:51.864938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.864826    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq" event={"ID":"4a675188-483b-4452-b0bb-e3456f958300","Type":"ContainerDied","Data":"a65233e441f25dde16de39dcaafb1fbc1a63b945870eda7184f004e59e631d32"}
Sep 08 17:05:51.864938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.864843    2542 scope.go:117] "RemoveContainer" containerID="091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"
Sep 08 17:05:51.864938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.864795    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"
Sep 08 17:05:51.874837 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.874816    2542 scope.go:117] "RemoveContainer" containerID="091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"
Sep 08 17:05:51.875119 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:05:51.875100    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3\": container with ID starting with 091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3 not found: ID does not exist" containerID="091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"
Sep 08 17:05:51.875180 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.875130    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3"} err="failed to get container status \"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3\": rpc error: code = NotFound desc = could not find container \"091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3\": container with ID starting with 091a6da469e09aaf7c3e5f128484baa47152bb3b4370868a298980c9e880c4f3 not found: ID does not exist"
Sep 08 17:05:51.887407 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.887383    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:51.888878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.888859    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-66e7a8/autores-model-66e7a8-kserve-547768fb55-55vbq"]
Sep 08 17:05:51.940655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.940615    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.940655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.940644    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a675188-483b-4452-b0bb-e3456f958300-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.940655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.940655    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a675188-483b-4452-b0bb-e3456f958300-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:51.940894 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:51.940668    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz54t\" (UniqueName: \"kubernetes.io/projected/4a675188-483b-4452-b0bb-e3456f958300-kube-api-access-mz54t\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:05:53.116598 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:53.116565    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a675188-483b-4452-b0bb-e3456f958300" path="/var/lib/kubelet/pods/4a675188-483b-4452-b0bb-e3456f958300/volumes"
Sep 08 17:05:56.990987 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:56.990962    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:05:56.992558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:05:56.992541    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:06:00.999728 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:00.999689    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000010    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a675188-483b-4452-b0bb-e3456f958300" containerName="main"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000021    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a675188-483b-4452-b0bb-e3456f958300" containerName="main"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000032    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f487ed-9325-4725-b655-ee2d60bff056" containerName="payload-processing"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000038    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f487ed-9325-4725-b655-ee2d60bff056" containerName="payload-processing"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000047    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a404aa-1491-4a23-8a44-7a9521ba026f" containerName="main"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000052    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a404aa-1491-4a23-8a44-7a9521ba026f" containerName="main"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000061    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" containerName="payload-pre-processing"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000067    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" containerName="payload-pre-processing"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000078    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c97a36-e10a-4a27-98f7-aaa420025aa9" containerName="revoke-keys"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000085    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c97a36-e10a-4a27-98f7-aaa420025aa9" containerName="revoke-keys"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000093    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" containerName="main"
Sep 08 17:06:01.000119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000099    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" containerName="main"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000163    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a675188-483b-4452-b0bb-e3456f958300" containerName="main"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000171    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9facd4-48a9-47c6-9177-ec4db9fd9251" containerName="payload-pre-processing"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000177    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c97a36-e10a-4a27-98f7-aaa420025aa9" containerName="revoke-keys"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000182    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f487ed-9325-4725-b655-ee2d60bff056" containerName="payload-processing"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000188    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="be590c0c-d293-40f4-a76a-e5e6ad70b4fe" containerName="main"
Sep 08 17:06:01.000554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.000193    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a404aa-1491-4a23-8a44-7a9521ba026f" containerName="main"
Sep 08 17:06:01.004512 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.004495    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.010463 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.010437    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3b5d3c-dockercfg-k5pcj\""
Sep 08 17:06:01.010603 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.010485    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3b5d3c\""
Sep 08 17:06:01.022559 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.022482    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:06:01.118601 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.118566    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.118774 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.118633    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkzk6\" (UniqueName: \"kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.219580 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.219537    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.219780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.219613    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkzk6\" (UniqueName: \"kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.220275 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.220250    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.239648 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.239617    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkzk6\" (UniqueName: \"kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6\") pod \"payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") " pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.315354 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.315253    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:01.597841 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.597780    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:06:01.600538 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:06:01.600511    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160c1c6a_9db9_4947_a2fb_8dc97a62119e.slice/crio-110ee7ca849e6226751e70ea153a9cbf36584062953254de375a0d2fcf1c633d WatchSource:0}: Error finding container 110ee7ca849e6226751e70ea153a9cbf36584062953254de375a0d2fcf1c633d: Status 404 returned error can't find the container with id 110ee7ca849e6226751e70ea153a9cbf36584062953254de375a0d2fcf1c633d
Sep 08 17:06:01.914433 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.914332    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" event={"ID":"160c1c6a-9db9-4947-a2fb-8dc97a62119e","Type":"ContainerStarted","Data":"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"}
Sep 08 17:06:01.914433 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.914396    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" event={"ID":"160c1c6a-9db9-4947-a2fb-8dc97a62119e","Type":"ContainerStarted","Data":"110ee7ca849e6226751e70ea153a9cbf36584062953254de375a0d2fcf1c633d"}
Sep 08 17:06:01.914637 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:01.914524    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:11.814729 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.814669    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" podStartSLOduration=11.814651455 podStartE2EDuration="11.814651455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:01.954966151 +0000 UTC m=+905.422172343" watchObservedRunningTime="2026-09-08 17:06:11.814651455 +0000 UTC m=+915.281857649"
Sep 08 17:06:11.816594 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.816568    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:06:11.820132 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.820111    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.825238 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.825217    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a7cab\"/\"openshift-service-ca.crt\""
Sep 08 17:06:11.825398 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.825365    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a7cab\"/\"default-dockercfg-96s9z\""
Sep 08 17:06:11.826572 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.826505    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a7cab\"/\"kube-root-ca.crt\""
Sep 08 17:06:11.826636 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.826590    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a7cab\"/\"test-model-5a7cab-kserve-self-signed-certs\""
Sep 08 17:06:11.831533 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.831513    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:06:11.920557 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920525    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.920557 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920562    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj58g\" (UniqueName: \"kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.920760 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920654    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.920760 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920699    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.920760 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920723    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:11.920760 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:11.920744    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022111 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022069    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022111 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022114    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022132    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj58g\" (UniqueName: \"kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022208    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022268    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022290    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022650 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022625    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022726 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022658    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.022726 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.022705    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.024364 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.024342    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.024602 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.024585    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.030922 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.030902    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj58g\" (UniqueName: \"kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g\") pod \"test-model-5a7cab-kserve-78887c6987-4lwr9\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") " pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.129438 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.129353    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:12.401536 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.401508    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:06:12.403303 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:06:12.403275    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57a2eb95_5967_486d_a727_275eb0e210db.slice/crio-6e270964607cf66f33a6ed042074b99a1716feb9e2c27407f2d8df3ce46570cd WatchSource:0}: Error finding container 6e270964607cf66f33a6ed042074b99a1716feb9e2c27407f2d8df3ce46570cd: Status 404 returned error can't find the container with id 6e270964607cf66f33a6ed042074b99a1716feb9e2c27407f2d8df3ce46570cd
Sep 08 17:06:12.921517 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.921481    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:12.967766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.967731    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" event={"ID":"57a2eb95-5967-486d-a727-275eb0e210db","Type":"ContainerStarted","Data":"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"}
Sep 08 17:06:12.967946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.967778    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" event={"ID":"57a2eb95-5967-486d-a727-275eb0e210db","Type":"ContainerStarted","Data":"6e270964607cf66f33a6ed042074b99a1716feb9e2c27407f2d8df3ce46570cd"}
Sep 08 17:06:12.984697 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:12.984645    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" podStartSLOduration=1.9846294169999998 podStartE2EDuration="1.984629417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:12.982814811 +0000 UTC m=+916.450021003" watchObservedRunningTime="2026-09-08 17:06:12.984629417 +0000 UTC m=+916.451835609"
Sep 08 17:06:22.129612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:22.129550    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:22.129612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:22.129626    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:22.142356 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:22.142314    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:23.026918 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:23.026886    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:06:36.122098 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.122067    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:06:36.125426 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.125411    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.128310 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.128287    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3b5d3c\"/\"openshift-service-ca.crt\""
Sep 08 17:06:36.129390 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129363    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3b5d3c\"/\"default-dockercfg-c7vn4\""
Sep 08 17:06:36.129511 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129451    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3b5d3c\"/\"kube-root-ca.crt\""
Sep 08 17:06:36.129511 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129487    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3b5d3c\"/\"explicit-model-3b5d3c-kserve-self-signed-certs\""
Sep 08 17:06:36.129837 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129816    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.129881 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129859    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.129981 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.129958    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8lw\" (UniqueName: \"kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.130047 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.130003    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.130047 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.130028    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.130144 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.130065    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.138161 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.138135    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:06:36.230983 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.230949    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231000    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8lw\" (UniqueName: \"kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231019    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231037    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231057    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231101    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231455 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231426    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231541 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231506    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.231592 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.231544    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.233371 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.233349    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.233555 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.233537    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.239506 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.239474    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8lw\" (UniqueName: \"kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw\") pod \"explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") " pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.435262 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.435231    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:36.708276 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:36.708253    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:06:36.710936 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:06:36.710903    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78a9cbd2_150a_43a6_b017_97ef4a945565.slice/crio-ab698cafeb9d49163b1c60ff912bfaee0ffb47c014a0127f493042d6bf7ffa12 WatchSource:0}: Error finding container ab698cafeb9d49163b1c60ff912bfaee0ffb47c014a0127f493042d6bf7ffa12: Status 404 returned error can't find the container with id ab698cafeb9d49163b1c60ff912bfaee0ffb47c014a0127f493042d6bf7ffa12
Sep 08 17:06:37.085861 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:37.085771    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" event={"ID":"78a9cbd2-150a-43a6-b017-97ef4a945565","Type":"ContainerStarted","Data":"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"}
Sep 08 17:06:37.085861 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:37.085815    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" event={"ID":"78a9cbd2-150a-43a6-b017-97ef4a945565","Type":"ContainerStarted","Data":"ab698cafeb9d49163b1c60ff912bfaee0ffb47c014a0127f493042d6bf7ffa12"}
Sep 08 17:06:37.103027 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:37.102978    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" podStartSLOduration=1.102960452 podStartE2EDuration="1.102960452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:06:37.101929588 +0000 UTC m=+940.569135783" watchObservedRunningTime="2026-09-08 17:06:37.102960452 +0000 UTC m=+940.570166636"
Sep 08 17:06:46.436400 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:46.436364    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:46.436803 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:46.436417    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:46.448592 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:46.448563    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:47.141477 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:47.141444    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:06:47.463673 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:47.463639    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:06:48.220740 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:48.220706    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:48.220910 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:48.220780    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:06:48.720763904 +0000 UTC m=+952.187970074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:48.725678 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:48.725640    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:48.726153 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:48.725735    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:06:49.725715416 +0000 UTC m=+953.192921591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:49.735456 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:49.735421    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:49.735925 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:49.735504    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:06:51.7354847 +0000 UTC m=+955.202690872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:51.755303 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:51.755263    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:51.755712 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:51.755362    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:06:55.755345829 +0000 UTC m=+959.222552012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:55.790619 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:55.790585    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:55.790998 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:55.790662    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:07:03.79064801 +0000 UTC m=+967.257854181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:06:58.227639 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.227595    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:06:58.228046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.227853    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" podUID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" containerName="payload-processing" containerID="cri-o://15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4" gracePeriod=30
Sep 08 17:06:58.478584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.478524    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:58.512548 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.512512    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkzk6\" (UniqueName: \"kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6\") pod \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") "
Sep 08 17:06:58.512730 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.512623    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume\") pod \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\" (UID: \"160c1c6a-9db9-4947-a2fb-8dc97a62119e\") "
Sep 08 17:06:58.512973 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.512951    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "160c1c6a-9db9-4947-a2fb-8dc97a62119e" (UID: "160c1c6a-9db9-4947-a2fb-8dc97a62119e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:06:58.514777 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.514753    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6" (OuterVolumeSpecName: "kube-api-access-vkzk6") pod "160c1c6a-9db9-4947-a2fb-8dc97a62119e" (UID: "160c1c6a-9db9-4947-a2fb-8dc97a62119e"). InnerVolumeSpecName "kube-api-access-vkzk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:06:58.613317 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.613278    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160c1c6a-9db9-4947-a2fb-8dc97a62119e-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:06:58.613317 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:58.613308    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkzk6\" (UniqueName: \"kubernetes.io/projected/160c1c6a-9db9-4947-a2fb-8dc97a62119e-kube-api-access-vkzk6\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:06:59.190579 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.190546    2542 generic.go:356] "Generic (PLEG): container finished" podID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" containerID="15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4" exitCode=0
Sep 08 17:06:59.190773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.190589    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"
Sep 08 17:06:59.190773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.190593    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" event={"ID":"160c1c6a-9db9-4947-a2fb-8dc97a62119e","Type":"ContainerDied","Data":"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"}
Sep 08 17:06:59.190773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.190730    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db" event={"ID":"160c1c6a-9db9-4947-a2fb-8dc97a62119e","Type":"ContainerDied","Data":"110ee7ca849e6226751e70ea153a9cbf36584062953254de375a0d2fcf1c633d"}
Sep 08 17:06:59.190773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.190768    2542 scope.go:117] "RemoveContainer" containerID="15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"
Sep 08 17:06:59.199518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.199494    2542 scope.go:117] "RemoveContainer" containerID="15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"
Sep 08 17:06:59.199794 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:06:59.199774    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4\": container with ID starting with 15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4 not found: ID does not exist" containerID="15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"
Sep 08 17:06:59.199851 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.199804    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4"} err="failed to get container status \"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4\": rpc error: code = NotFound desc = could not find container \"15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4\": container with ID starting with 15ee7e5a5552904285f46fc7477d1056bc782ae4a1efd887baffdf00a7900ff4 not found: ID does not exist"
Sep 08 17:06:59.214078 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.214051    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:06:59.221091 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:06:59.221066    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3b5d3c-576cf94949-5f9db"]
Sep 08 17:07:01.122510 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:01.120314    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" path="/var/lib/kubelet/pods/160c1c6a-9db9-4947-a2fb-8dc97a62119e/volumes"
Sep 08 17:07:03.861138 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:03.861098    2542 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-self-signed-certs: secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:07:03.861654 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:03.861183    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs podName:78a9cbd2-150a-43a6-b017-97ef4a945565 nodeName:}" failed. No retries permitted until 2026-09-08 17:07:19.861167062 +0000 UTC m=+983.328373233 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs") pod "explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565") : secret "explicit-model-3b5d3c-kserve-self-signed-certs" not found
Sep 08 17:07:04.159846 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.159762    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" podUID="78a9cbd2-150a-43a6-b017-97ef4a945565" containerName="main" containerID="cri-o://cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12" gracePeriod=45
Sep 08 17:07:04.404813 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.404790    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:07:04.467883 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.467855    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv8lw\" (UniqueName: \"kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.467951    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.467971    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.467999    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.468029    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468259 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.468086    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home\") pod \"78a9cbd2-150a-43a6-b017-97ef4a945565\" (UID: \"78a9cbd2-150a-43a6-b017-97ef4a945565\") "
Sep 08 17:07:04.468316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.468258    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:04.468398 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.468310    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache" (OuterVolumeSpecName: "model-cache") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:04.468473 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.468446    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home" (OuterVolumeSpecName: "home") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:04.470207 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.470184    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm" (OuterVolumeSpecName: "dshm") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:04.470207 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.470199    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw" (OuterVolumeSpecName: "kube-api-access-mv8lw") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "kube-api-access-mv8lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:07:04.470313 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.470189    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78a9cbd2-150a-43a6-b017-97ef4a945565" (UID: "78a9cbd2-150a-43a6-b017-97ef4a945565"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:07:04.569101 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569064    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv8lw\" (UniqueName: \"kubernetes.io/projected/78a9cbd2-150a-43a6-b017-97ef4a945565-kube-api-access-mv8lw\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:04.569101 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569096    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:04.569101 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569108    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:04.569374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569119    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:04.569374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569127    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78a9cbd2-150a-43a6-b017-97ef4a945565-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:04.569374 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:04.569134    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78a9cbd2-150a-43a6-b017-97ef4a945565-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:05.218910 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.218873    2542 generic.go:356] "Generic (PLEG): container finished" podID="78a9cbd2-150a-43a6-b017-97ef4a945565" containerID="cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12" exitCode=0
Sep 08 17:07:05.218910 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.218913    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"
Sep 08 17:07:05.219401 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.218924    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" event={"ID":"78a9cbd2-150a-43a6-b017-97ef4a945565","Type":"ContainerDied","Data":"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"}
Sep 08 17:07:05.219401 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.218956    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q" event={"ID":"78a9cbd2-150a-43a6-b017-97ef4a945565","Type":"ContainerDied","Data":"ab698cafeb9d49163b1c60ff912bfaee0ffb47c014a0127f493042d6bf7ffa12"}
Sep 08 17:07:05.219401 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.218977    2542 scope.go:117] "RemoveContainer" containerID="cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"
Sep 08 17:07:05.227078 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.227059    2542 scope.go:117] "RemoveContainer" containerID="cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"
Sep 08 17:07:05.227361 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:05.227341    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12\": container with ID starting with cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12 not found: ID does not exist" containerID="cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"
Sep 08 17:07:05.227432 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.227371    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12"} err="failed to get container status \"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12\": rpc error: code = NotFound desc = could not find container \"cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12\": container with ID starting with cd53b6662a9818615fb0f657dc833b183ecece69f5a1abafa981267bce184f12 not found: ID does not exist"
Sep 08 17:07:05.252985 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.252958    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:07:05.260769 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:05.260748    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3b5d3c/explicit-model-3b5d3c-kserve-cb4c6898f-qpc4q"]
Sep 08 17:07:07.116372 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:07.116339    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a9cbd2-150a-43a6-b017-97ef4a945565" path="/var/lib/kubelet/pods/78a9cbd2-150a-43a6-b017-97ef4a945565/volumes"
Sep 08 17:07:18.702547 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702514    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:07:18.702930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702843    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a9cbd2-150a-43a6-b017-97ef4a945565" containerName="main"
Sep 08 17:07:18.702930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702853    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a9cbd2-150a-43a6-b017-97ef4a945565" containerName="main"
Sep 08 17:07:18.702930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702862    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" containerName="payload-processing"
Sep 08 17:07:18.702930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702867    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" containerName="payload-processing"
Sep 08 17:07:18.702930 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702930    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a9cbd2-150a-43a6-b017-97ef4a945565" containerName="main"
Sep 08 17:07:18.703112 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.702942    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="160c1c6a-9db9-4947-a2fb-8dc97a62119e" containerName="payload-processing"
Sep 08 17:07:18.707192 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.707165    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.711027 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.711000    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-79e919\"/\"openshift-service-ca.crt\""
Sep 08 17:07:18.711170 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.711027    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-79e919\"/\"kube-root-ca.crt\""
Sep 08 17:07:18.711170 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.711055    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-79e919\"/\"default-dockercfg-ntxh9\""
Sep 08 17:07:18.711170 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.711000    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-79e919\"/\"noten-model-79e919-kserve-self-signed-certs\""
Sep 08 17:07:18.717536 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.717514    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:07:18.796500 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796467    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.796500 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796508    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.796698 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796527    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.796698 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796647    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.796698 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796680    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcnr\" (UniqueName: \"kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.796796 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.796763    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897183 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897141    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897205    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897228    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897245    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897284    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897311    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcnr\" (UniqueName: \"kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897687 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897599    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897687 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897628    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.897815 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.897722    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.899488 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.899459    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.899710 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.899692    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:18.906000 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:18.905969    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcnr\" (UniqueName: \"kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr\") pod \"noten-model-79e919-kserve-848cfc87b7-t5r5h\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") " pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:19.017535 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:19.017408    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:19.293035 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:19.293012    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:07:19.297211 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:07:19.297185    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73333ef0_027d_4668_833f_3bde0385152a.slice/crio-61c0bed303d61776fe6e40b0591d23af24ee3297794aa9bbcf3f82337b281160 WatchSource:0}: Error finding container 61c0bed303d61776fe6e40b0591d23af24ee3297794aa9bbcf3f82337b281160: Status 404 returned error can't find the container with id 61c0bed303d61776fe6e40b0591d23af24ee3297794aa9bbcf3f82337b281160
Sep 08 17:07:20.291928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:20.291892    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" event={"ID":"73333ef0-027d-4668-833f-3bde0385152a","Type":"ContainerStarted","Data":"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"}
Sep 08 17:07:20.291928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:20.291934    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" event={"ID":"73333ef0-027d-4668-833f-3bde0385152a","Type":"ContainerStarted","Data":"61c0bed303d61776fe6e40b0591d23af24ee3297794aa9bbcf3f82337b281160"}
Sep 08 17:07:20.310372 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:20.310300    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" podStartSLOduration=2.31028463 podStartE2EDuration="2.31028463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:20.308961937 +0000 UTC m=+983.776168130" watchObservedRunningTime="2026-09-08 17:07:20.31028463 +0000 UTC m=+983.777490823"
Sep 08 17:07:20.751863 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:20.751826    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:07:27.258115 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.258084    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:07:27.261513 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.261495    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.265931 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.265908    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a837a8a9\""
Sep 08 17:07:27.266035 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.265962    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a837a8a9-dockercfg-42h4k\""
Sep 08 17:07:27.275949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.275920    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:07:27.374607 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.374573    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.374607 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.374608    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshnk\" (UniqueName: \"kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.475611 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.475569    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.475611 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.475613    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshnk\" (UniqueName: \"kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.476252 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.476232    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.484352 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.484303    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshnk\" (UniqueName: \"kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk\") pod \"payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") " pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.572454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.572371    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:27.848522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:27.848445    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:07:27.851850 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:07:27.851824    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e4544d_205b_46ec_941c_bb3d4088e1fd.slice/crio-0222995a91a54ab2b16259fd486e54e04e649caeabf8d9dc4c4e06ec0c892f98 WatchSource:0}: Error finding container 0222995a91a54ab2b16259fd486e54e04e649caeabf8d9dc4c4e06ec0c892f98: Status 404 returned error can't find the container with id 0222995a91a54ab2b16259fd486e54e04e649caeabf8d9dc4c4e06ec0c892f98
Sep 08 17:07:28.330970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:28.330928    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" event={"ID":"12e4544d-205b-46ec-941c-bb3d4088e1fd","Type":"ContainerStarted","Data":"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"}
Sep 08 17:07:28.331471 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:28.330987    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" event={"ID":"12e4544d-205b-46ec-941c-bb3d4088e1fd","Type":"ContainerStarted","Data":"0222995a91a54ab2b16259fd486e54e04e649caeabf8d9dc4c4e06ec0c892f98"}
Sep 08 17:07:28.331471 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:28.331008    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:28.352911 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:28.352857    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" podStartSLOduration=1.352842152 podStartE2EDuration="1.352842152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:07:28.350922072 +0000 UTC m=+991.818128267" watchObservedRunningTime="2026-09-08 17:07:28.352842152 +0000 UTC m=+991.820048345"
Sep 08 17:07:29.018209 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:29.018168    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:29.018388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:29.018257    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:29.031075 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:29.031047    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:29.347426 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:29.347318    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:07:35.771364 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:35.771283    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" podUID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" containerName="main" containerID="cri-o://cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f" gracePeriod=45
Sep 08 17:07:36.014977 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.014952    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:07:36.040897 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.040823    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.040897 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.040879    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.041117 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.040930    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.041117 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.040952    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.041117 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.041094    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:36.041267 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.041244    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache" (OuterVolumeSpecName: "model-cache") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:36.041267 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.041255    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.043238 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.043205    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:07:36.043238 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.043212    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm" (OuterVolumeSpecName: "dshm") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:36.142404 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142369    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vqzp\" (UniqueName: \"kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.142404 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142405    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home\") pod \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\" (UID: \"8219fc56-185f-4b3d-bf24-8a53a39a4e58\") "
Sep 08 17:07:36.142604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142577    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8219fc56-185f-4b3d-bf24-8a53a39a4e58-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.142604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142588    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.142604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142597    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.142727 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.142700    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home" (OuterVolumeSpecName: "home") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:07:36.144436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.144416    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp" (OuterVolumeSpecName: "kube-api-access-7vqzp") pod "8219fc56-185f-4b3d-bf24-8a53a39a4e58" (UID: "8219fc56-185f-4b3d-bf24-8a53a39a4e58"). InnerVolumeSpecName "kube-api-access-7vqzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:07:36.244040 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.243999    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqzp\" (UniqueName: \"kubernetes.io/projected/8219fc56-185f-4b3d-bf24-8a53a39a4e58-kube-api-access-7vqzp\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.244040 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.244032    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8219fc56-185f-4b3d-bf24-8a53a39a4e58-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:36.370226 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.370141    2542 generic.go:356] "Generic (PLEG): container finished" podID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" containerID="cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f" exitCode=0
Sep 08 17:07:36.370226 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.370180    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"
Sep 08 17:07:36.370425 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.370226    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" event={"ID":"8219fc56-185f-4b3d-bf24-8a53a39a4e58","Type":"ContainerDied","Data":"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"}
Sep 08 17:07:36.370425 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.370268    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5" event={"ID":"8219fc56-185f-4b3d-bf24-8a53a39a4e58","Type":"ContainerDied","Data":"f3f51dd37931a92f2d138b4c7ae76b12ef38c42496e361f3008eea57efdebe3c"}
Sep 08 17:07:36.370425 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.370288    2542 scope.go:117] "RemoveContainer" containerID="cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"
Sep 08 17:07:36.388300 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.388276    2542 scope.go:117] "RemoveContainer" containerID="cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"
Sep 08 17:07:36.388632 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:36.388614    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f\": container with ID starting with cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f not found: ID does not exist" containerID="cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"
Sep 08 17:07:36.388697 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.388642    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f"} err="failed to get container status \"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f\": rpc error: code = NotFound desc = could not find container \"cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f\": container with ID starting with cdaadb00235df9e25ce5a503781c29064ba9118961692485f2fb1d082e72115f not found: ID does not exist"
Sep 08 17:07:36.397036 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.397012    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:07:36.401057 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:36.401033    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2746ad/test-model-2746ad-kserve-6998d755d9-xd7k5"]
Sep 08 17:07:37.116483 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:37.116407    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" path="/var/lib/kubelet/pods/8219fc56-185f-4b3d-bf24-8a53a39a4e58/volumes"
Sep 08 17:07:39.338441 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:39.338409    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:07:48.419343 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:48.419296    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:07:49.089286 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:49.089251    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:07:58.876121 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:58.876081    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:07:58.876556 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:58.876337    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" podUID="56281f02-3298-4744-9a0b-4d278a5e4f70" containerName="payload-processing" containerID="cri-o://e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e" gracePeriod=30
Sep 08 17:07:58.910876 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:58.910842    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:07:58.911134 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:58.911080    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" podUID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" containerName="payload-pre-processing" containerID="cri-o://f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548" gracePeriod=30
Sep 08 17:07:59.187495 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.187467    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:07:59.190098 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.190079    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:07:59.243126 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243094    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqnjp\" (UniqueName: \"kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp\") pod \"56281f02-3298-4744-9a0b-4d278a5e4f70\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") "
Sep 08 17:07:59.243315 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243136    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume\") pod \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") "
Sep 08 17:07:59.243315 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243174    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume\") pod \"56281f02-3298-4744-9a0b-4d278a5e4f70\" (UID: \"56281f02-3298-4744-9a0b-4d278a5e4f70\") "
Sep 08 17:07:59.243315 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243204    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdlvs\" (UniqueName: \"kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs\") pod \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\" (UID: \"c370cf4c-0c68-471c-91c1-e3b61c9fb769\") "
Sep 08 17:07:59.243595 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243566    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c370cf4c-0c68-471c-91c1-e3b61c9fb769" (UID: "c370cf4c-0c68-471c-91c1-e3b61c9fb769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:07:59.243720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.243610    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56281f02-3298-4744-9a0b-4d278a5e4f70" (UID: "56281f02-3298-4744-9a0b-4d278a5e4f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:07:59.245277 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.245257    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs" (OuterVolumeSpecName: "kube-api-access-pdlvs") pod "c370cf4c-0c68-471c-91c1-e3b61c9fb769" (UID: "c370cf4c-0c68-471c-91c1-e3b61c9fb769"). InnerVolumeSpecName "kube-api-access-pdlvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:07:59.245346 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.245288    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp" (OuterVolumeSpecName: "kube-api-access-cqnjp") pod "56281f02-3298-4744-9a0b-4d278a5e4f70" (UID: "56281f02-3298-4744-9a0b-4d278a5e4f70"). InnerVolumeSpecName "kube-api-access-cqnjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:07:59.344516 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.344481    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqnjp\" (UniqueName: \"kubernetes.io/projected/56281f02-3298-4744-9a0b-4d278a5e4f70-kube-api-access-cqnjp\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:59.344516 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.344514    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c370cf4c-0c68-471c-91c1-e3b61c9fb769-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:59.344715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.344529    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56281f02-3298-4744-9a0b-4d278a5e4f70-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:59.344715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.344544    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlvs\" (UniqueName: \"kubernetes.io/projected/c370cf4c-0c68-471c-91c1-e3b61c9fb769-kube-api-access-pdlvs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:07:59.475448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.475409    2542 generic.go:356] "Generic (PLEG): container finished" podID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" containerID="f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548" exitCode=0
Sep 08 17:07:59.475610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.475456    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"
Sep 08 17:07:59.475610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.475493    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" event={"ID":"c370cf4c-0c68-471c-91c1-e3b61c9fb769","Type":"ContainerDied","Data":"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"}
Sep 08 17:07:59.475610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.475547    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv" event={"ID":"c370cf4c-0c68-471c-91c1-e3b61c9fb769","Type":"ContainerDied","Data":"fbc37610f67d3e1ad0a120a7525e0898eb0be24959ab5367207f2ec3c172c019"}
Sep 08 17:07:59.475610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.475575    2542 scope.go:117] "RemoveContainer" containerID="f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"
Sep 08 17:07:59.476879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.476861    2542 generic.go:356] "Generic (PLEG): container finished" podID="56281f02-3298-4744-9a0b-4d278a5e4f70" containerID="e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e" exitCode=0
Sep 08 17:07:59.476961 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.476897    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"
Sep 08 17:07:59.476961 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.476915    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" event={"ID":"56281f02-3298-4744-9a0b-4d278a5e4f70","Type":"ContainerDied","Data":"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"}
Sep 08 17:07:59.476961 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.476954    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28" event={"ID":"56281f02-3298-4744-9a0b-4d278a5e4f70","Type":"ContainerDied","Data":"76c8537c420c23b1af8c97a251171d00d93beeaaf85fdb3d3b79a61cbd9d9cfc"}
Sep 08 17:07:59.487527 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.487507    2542 scope.go:117] "RemoveContainer" containerID="f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"
Sep 08 17:07:59.487860 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:59.487840    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548\": container with ID starting with f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548 not found: ID does not exist" containerID="f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"
Sep 08 17:07:59.487945 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.487870    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548"} err="failed to get container status \"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548\": rpc error: code = NotFound desc = could not find container \"f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548\": container with ID starting with f009af4d611019c3536b6ecb3f7fc82e87ec47d0e12fe863dd391e89b6f8a548 not found: ID does not exist"
Sep 08 17:07:59.487945 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.487893    2542 scope.go:117] "RemoveContainer" containerID="e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"
Sep 08 17:07:59.495365 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.495350    2542 scope.go:117] "RemoveContainer" containerID="e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"
Sep 08 17:07:59.495614 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:07:59.495597    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e\": container with ID starting with e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e not found: ID does not exist" containerID="e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"
Sep 08 17:07:59.495673 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.495624    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e"} err="failed to get container status \"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e\": rpc error: code = NotFound desc = could not find container \"e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e\": container with ID starting with e4eec56853016ca92d262218e5b08156989d4e69fcf7bac0153bcf773402321e not found: ID does not exist"
Sep 08 17:07:59.501124 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.501099    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:07:59.505042 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.505022    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2746ad-69556fb866-hrpqv"]
Sep 08 17:07:59.517038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.517010    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:07:59.522191 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:07:59.522167    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2746ad-c4dd789cc-xmc28"]
Sep 08 17:08:01.116861 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:01.116828    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56281f02-3298-4744-9a0b-4d278a5e4f70" path="/var/lib/kubelet/pods/56281f02-3298-4744-9a0b-4d278a5e4f70/volumes"
Sep 08 17:08:01.117224 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:01.117163    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" path="/var/lib/kubelet/pods/c370cf4c-0c68-471c-91c1-e3b61c9fb769/volumes"
Sep 08 17:08:03.438819 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.438774    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" podUID="57a2eb95-5967-486d-a727-275eb0e210db" containerName="main" containerID="cri-o://b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41" gracePeriod=45
Sep 08 17:08:03.684608 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.684583    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:08:03.779472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779379    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj58g\" (UniqueName: \"kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779419    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779448    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779472    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779784 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779534    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779784 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779572    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm\") pod \"57a2eb95-5967-486d-a727-275eb0e210db\" (UID: \"57a2eb95-5967-486d-a727-275eb0e210db\") "
Sep 08 17:08:03.779784 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779716    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache" (OuterVolumeSpecName: "model-cache") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:03.779784 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779760    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home" (OuterVolumeSpecName: "home") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:03.779988 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779906    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:03.779988 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779948    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:03.779988 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.779964    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:03.781874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.781847    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm" (OuterVolumeSpecName: "dshm") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:03.781991 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.781846    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g" (OuterVolumeSpecName: "kube-api-access-wj58g") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "kube-api-access-wj58g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:08:03.781991 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.781876    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57a2eb95-5967-486d-a727-275eb0e210db" (UID: "57a2eb95-5967-486d-a727-275eb0e210db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:08:03.881341 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.881301    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:03.881509 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.881355    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a2eb95-5967-486d-a727-275eb0e210db-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:03.881509 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.881365    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj58g\" (UniqueName: \"kubernetes.io/projected/57a2eb95-5967-486d-a727-275eb0e210db-kube-api-access-wj58g\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:03.881509 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:03.881375    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a2eb95-5967-486d-a727-275eb0e210db-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.109665 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.109548    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" podUID="73333ef0-027d-4668-833f-3bde0385152a" containerName="main" containerID="cri-o://232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab" gracePeriod=45
Sep 08 17:08:04.350116 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.350086    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:08:04.486650 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486622    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486657    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwcnr\" (UniqueName: \"kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486704    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486719    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486747    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486780    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home\") pod \"73333ef0-027d-4668-833f-3bde0385152a\" (UID: \"73333ef0-027d-4668-833f-3bde0385152a\") "
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486973    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:04.487113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.486984    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache" (OuterVolumeSpecName: "model-cache") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:04.487450 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.487133    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home" (OuterVolumeSpecName: "home") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:04.488931 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.488900    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm" (OuterVolumeSpecName: "dshm") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:08:04.489034 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.488985    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr" (OuterVolumeSpecName: "kube-api-access-gwcnr") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "kube-api-access-gwcnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:08:04.489034 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.489015    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73333ef0-027d-4668-833f-3bde0385152a" (UID: "73333ef0-027d-4668-833f-3bde0385152a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:08:04.503650 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.503626    2542 generic.go:356] "Generic (PLEG): container finished" podID="73333ef0-027d-4668-833f-3bde0385152a" containerID="232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab" exitCode=0
Sep 08 17:08:04.503766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.503670    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"
Sep 08 17:08:04.503766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.503712    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" event={"ID":"73333ef0-027d-4668-833f-3bde0385152a","Type":"ContainerDied","Data":"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"}
Sep 08 17:08:04.503766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.503756    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h" event={"ID":"73333ef0-027d-4668-833f-3bde0385152a","Type":"ContainerDied","Data":"61c0bed303d61776fe6e40b0591d23af24ee3297794aa9bbcf3f82337b281160"}
Sep 08 17:08:04.503923 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.503777    2542 scope.go:117] "RemoveContainer" containerID="232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"
Sep 08 17:08:04.505141 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.505084    2542 generic.go:356] "Generic (PLEG): container finished" podID="57a2eb95-5967-486d-a727-275eb0e210db" containerID="b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41" exitCode=0
Sep 08 17:08:04.505141 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.505113    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" event={"ID":"57a2eb95-5967-486d-a727-275eb0e210db","Type":"ContainerDied","Data":"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"}
Sep 08 17:08:04.505268 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.505138    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"
Sep 08 17:08:04.505268 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.505156    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9" event={"ID":"57a2eb95-5967-486d-a727-275eb0e210db","Type":"ContainerDied","Data":"6e270964607cf66f33a6ed042074b99a1716feb9e2c27407f2d8df3ce46570cd"}
Sep 08 17:08:04.512067 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.512044    2542 scope.go:117] "RemoveContainer" containerID="232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"
Sep 08 17:08:04.512371 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:08:04.512348    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab\": container with ID starting with 232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab not found: ID does not exist" containerID="232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"
Sep 08 17:08:04.512443 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.512378    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab"} err="failed to get container status \"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab\": rpc error: code = NotFound desc = could not find container \"232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab\": container with ID starting with 232312b7d626c8f8f2e30ab1d3558c5349aa9fece0534830daf75bd4b2099eab not found: ID does not exist"
Sep 08 17:08:04.512443 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.512394    2542 scope.go:117] "RemoveContainer" containerID="b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"
Sep 08 17:08:04.520067 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.520051    2542 scope.go:117] "RemoveContainer" containerID="b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"
Sep 08 17:08:04.520333 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:08:04.520309    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41\": container with ID starting with b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41 not found: ID does not exist" containerID="b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"
Sep 08 17:08:04.520383 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.520354    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41"} err="failed to get container status \"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41\": rpc error: code = NotFound desc = could not find container \"b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41\": container with ID starting with b7fba5e8a317ade5f115ea7c363176cd38e224e0553cbe3e7c8fa8c7fc7dda41 not found: ID does not exist"
Sep 08 17:08:04.529447 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.529423    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:08:04.533465 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.533444    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-79e919/noten-model-79e919-kserve-848cfc87b7-t5r5h"]
Sep 08 17:08:04.544155 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.544130    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:08:04.548116 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.548093    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5a7cab/test-model-5a7cab-kserve-78887c6987-4lwr9"]
Sep 08 17:08:04.587759 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587735    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.587759 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587758    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcnr\" (UniqueName: \"kubernetes.io/projected/73333ef0-027d-4668-833f-3bde0385152a-kube-api-access-gwcnr\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.587902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587769    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.587902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587779    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.587902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587787    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73333ef0-027d-4668-833f-3bde0385152a-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:04.587902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:04.587794    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73333ef0-027d-4668-833f-3bde0385152a-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:05.117373 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.117307    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57a2eb95-5967-486d-a727-275eb0e210db" path="/var/lib/kubelet/pods/57a2eb95-5967-486d-a727-275eb0e210db/volumes"
Sep 08 17:08:05.117748 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.117734    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73333ef0-027d-4668-833f-3bde0385152a" path="/var/lib/kubelet/pods/73333ef0-027d-4668-833f-3bde0385152a/volumes"
Sep 08 17:08:05.962218 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962184    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"]
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962510    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73333ef0-027d-4668-833f-3bde0385152a" containerName="main"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962523    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="73333ef0-027d-4668-833f-3bde0385152a" containerName="main"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962533    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" containerName="payload-pre-processing"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962539    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" containerName="payload-pre-processing"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962546    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a2eb95-5967-486d-a727-275eb0e210db" containerName="main"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962552    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a2eb95-5967-486d-a727-275eb0e210db" containerName="main"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962564    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56281f02-3298-4744-9a0b-4d278a5e4f70" containerName="payload-processing"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962569    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="56281f02-3298-4744-9a0b-4d278a5e4f70" containerName="payload-processing"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962579    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" containerName="main"
Sep 08 17:08:05.962619 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962585    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" containerName="main"
Sep 08 17:08:05.963222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962632    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="73333ef0-027d-4668-833f-3bde0385152a" containerName="main"
Sep 08 17:08:05.963222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962641    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="56281f02-3298-4744-9a0b-4d278a5e4f70" containerName="payload-processing"
Sep 08 17:08:05.963222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962650    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="c370cf4c-0c68-471c-91c1-e3b61c9fb769" containerName="payload-pre-processing"
Sep 08 17:08:05.963222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962659    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a2eb95-5967-486d-a727-275eb0e210db" containerName="main"
Sep 08 17:08:05.963222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.962669    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="8219fc56-185f-4b3d-bf24-8a53a39a4e58" containerName="main"
Sep 08 17:08:05.967193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.967172    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:05.969949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.969928    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:08:05.969949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.969947    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:08:05.973899 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:05.973862    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"]
Sep 08 17:08:06.100955 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:06.100918    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz\" (UID: \"f98111aa-921d-402b-89a2-4878f4b7838a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:06.201521 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:06.201483    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz\" (UID: \"f98111aa-921d-402b-89a2-4878f4b7838a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:06.202146 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:06.202119    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz\" (UID: \"f98111aa-921d-402b-89a2-4878f4b7838a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:06.280080 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:06.279999    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:06.548278 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:06.548205    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"]
Sep 08 17:08:06.551707 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:08:06.551681    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98111aa_921d_402b_89a2_4878f4b7838a.slice/crio-b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24 WatchSource:0}: Error finding container b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24: Status 404 returned error can't find the container with id b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24
Sep 08 17:08:07.524206 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:07.524169    2542 generic.go:356] "Generic (PLEG): container finished" podID="f98111aa-921d-402b-89a2-4878f4b7838a" containerID="e181a95dcc9e2b4f8f563ed590c5346c0224ccfaccb1aa8d0d1676ccae28970e" exitCode=0
Sep 08 17:08:07.524786 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:07.524248    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz" event={"ID":"f98111aa-921d-402b-89a2-4878f4b7838a","Type":"ContainerDied","Data":"e181a95dcc9e2b4f8f563ed590c5346c0224ccfaccb1aa8d0d1676ccae28970e"}
Sep 08 17:08:07.524786 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:07.524296    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz" event={"ID":"f98111aa-921d-402b-89a2-4878f4b7838a","Type":"ContainerStarted","Data":"b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24"}
Sep 08 17:08:08.481240 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.481206    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:08:08.481494 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.481464    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" podUID="12e4544d-205b-46ec-941c-bb3d4088e1fd" containerName="payload-processing" containerID="cri-o://fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e" gracePeriod=30
Sep 08 17:08:08.669662 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.669624    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:08.735074 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.735012    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:08:08.822900 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.822865    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca\") pod \"f98111aa-921d-402b-89a2-4878f4b7838a\" (UID: \"f98111aa-921d-402b-89a2-4878f4b7838a\") "
Sep 08 17:08:08.823060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.822909    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume\") pod \"12e4544d-205b-46ec-941c-bb3d4088e1fd\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") "
Sep 08 17:08:08.823060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.822956    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zshnk\" (UniqueName: \"kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk\") pod \"12e4544d-205b-46ec-941c-bb3d4088e1fd\" (UID: \"12e4544d-205b-46ec-941c-bb3d4088e1fd\") "
Sep 08 17:08:08.823242 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.823213    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f98111aa-921d-402b-89a2-4878f4b7838a" (UID: "f98111aa-921d-402b-89a2-4878f4b7838a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:08:08.823345 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.823283    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12e4544d-205b-46ec-941c-bb3d4088e1fd" (UID: "12e4544d-205b-46ec-941c-bb3d4088e1fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:08:08.825112 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.825088    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk" (OuterVolumeSpecName: "kube-api-access-zshnk") pod "12e4544d-205b-46ec-941c-bb3d4088e1fd" (UID: "12e4544d-205b-46ec-941c-bb3d4088e1fd"). InnerVolumeSpecName "kube-api-access-zshnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:08:08.923858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.923817    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshnk\" (UniqueName: \"kubernetes.io/projected/12e4544d-205b-46ec-941c-bb3d4088e1fd-kube-api-access-zshnk\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:08.923858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.923845    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f98111aa-921d-402b-89a2-4878f4b7838a-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:08.923858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:08.923859    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12e4544d-205b-46ec-941c-bb3d4088e1fd-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:09.535244 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.535208    2542 generic.go:356] "Generic (PLEG): container finished" podID="12e4544d-205b-46ec-941c-bb3d4088e1fd" containerID="fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e" exitCode=0
Sep 08 17:08:09.535445 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.535254    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"
Sep 08 17:08:09.535445 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.535278    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" event={"ID":"12e4544d-205b-46ec-941c-bb3d4088e1fd","Type":"ContainerDied","Data":"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"}
Sep 08 17:08:09.535445 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.535319    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2" event={"ID":"12e4544d-205b-46ec-941c-bb3d4088e1fd","Type":"ContainerDied","Data":"0222995a91a54ab2b16259fd486e54e04e649caeabf8d9dc4c4e06ec0c892f98"}
Sep 08 17:08:09.535445 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.535367    2542 scope.go:117] "RemoveContainer" containerID="fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"
Sep 08 17:08:09.536797 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.536660    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"
Sep 08 17:08:09.536797 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.536662    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz" event={"ID":"f98111aa-921d-402b-89a2-4878f4b7838a","Type":"ContainerDied","Data":"b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24"}
Sep 08 17:08:09.536797 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.536706    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b66801b18be9e9dc43de4099c866375df0e33e2e11e0d0b546f1ff8a8408ee24"
Sep 08 17:08:09.543865 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.543848    2542 scope.go:117] "RemoveContainer" containerID="fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"
Sep 08 17:08:09.544129 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:08:09.544107    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e\": container with ID starting with fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e not found: ID does not exist" containerID="fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"
Sep 08 17:08:09.544178 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.544137    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e"} err="failed to get container status \"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e\": rpc error: code = NotFound desc = could not find container \"fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e\": container with ID starting with fd65e84cab5ff73169c90a9898844390581aac28a8be77b67d3724182590852e not found: ID does not exist"
Sep 08 17:08:09.568439 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.568411    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:08:09.569920 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:09.569901    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a837a8a9-5c49cf8657-lh5j2"]
Sep 08 17:08:11.117158 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:11.117120    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e4544d-205b-46ec-941c-bb3d4088e1fd" path="/var/lib/kubelet/pods/12e4544d-205b-46ec-941c-bb3d4088e1fd/volumes"
Sep 08 17:08:16.776071 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:16.776033    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:08:16.776565 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:16.776286    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" podUID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" containerName="payload-pre-processing" containerID="cri-o://339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c" gracePeriod=30
Sep 08 17:08:16.940752 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:16.940719    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"]
Sep 08 17:08:16.944161 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:16.944133    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5a7cab-xd9dz"]
Sep 08 17:08:17.027840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.027784    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:08:17.116662 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.116631    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98111aa-921d-402b-89a2-4878f4b7838a" path="/var/lib/kubelet/pods/f98111aa-921d-402b-89a2-4878f4b7838a/volumes"
Sep 08 17:08:17.195813 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.195775    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume\") pod \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") "
Sep 08 17:08:17.195983 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.195838    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49w45\" (UniqueName: \"kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45\") pod \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\" (UID: \"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69\") "
Sep 08 17:08:17.196160 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.196135    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" (UID: "33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:08:17.198016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.197996    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45" (OuterVolumeSpecName: "kube-api-access-49w45") pod "33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" (UID: "33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69"). InnerVolumeSpecName "kube-api-access-49w45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:08:17.296674 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.296593    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:17.296674 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.296620    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49w45\" (UniqueName: \"kubernetes.io/projected/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69-kube-api-access-49w45\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:08:17.574714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.574627    2542 generic.go:356] "Generic (PLEG): container finished" podID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" containerID="339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c" exitCode=0
Sep 08 17:08:17.574714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.574674    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"
Sep 08 17:08:17.574887 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.574677    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" event={"ID":"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69","Type":"ContainerDied","Data":"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"}
Sep 08 17:08:17.574887 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.574799    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws" event={"ID":"33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69","Type":"ContainerDied","Data":"e6b0a7bdc82f5dc4551d5c9fffef5d3e2ebc95beac3082b5105d11dd22dedd18"}
Sep 08 17:08:17.574887 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.574825    2542 scope.go:117] "RemoveContainer" containerID="339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"
Sep 08 17:08:17.583409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.583389    2542 scope.go:117] "RemoveContainer" containerID="339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"
Sep 08 17:08:17.583681 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:08:17.583663    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c\": container with ID starting with 339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c not found: ID does not exist" containerID="339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"
Sep 08 17:08:17.583735 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.583690    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c"} err="failed to get container status \"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c\": rpc error: code = NotFound desc = could not find container \"339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c\": container with ID starting with 339a04463109349271df0719eb5cf5478d3ad09fb7fac14f5d9fd06dbf66c62c not found: ID does not exist"
Sep 08 17:08:17.595199 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.595176    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:08:17.601201 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:17.601180    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5a7cab-79b5b5ddb-7hwws"]
Sep 08 17:08:19.115822 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:19.115789    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" path="/var/lib/kubelet/pods/33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69/volumes"
Sep 08 17:08:29.608895 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.608861    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609156    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98111aa-921d-402b-89a2-4878f4b7838a" containerName="revoke-keys"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609168    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98111aa-921d-402b-89a2-4878f4b7838a" containerName="revoke-keys"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609175    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" containerName="payload-pre-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609181    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" containerName="payload-pre-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609204    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e4544d-205b-46ec-941c-bb3d4088e1fd" containerName="payload-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609210    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e4544d-205b-46ec-941c-bb3d4088e1fd" containerName="payload-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609255    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="33eecd8f-2c2e-45a1-aaf4-6ec89dc21a69" containerName="payload-pre-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609263    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e4544d-205b-46ec-941c-bb3d4088e1fd" containerName="payload-processing"
Sep 08 17:08:29.609274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.609270    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98111aa-921d-402b-89a2-4878f4b7838a" containerName="revoke-keys"
Sep 08 17:08:29.612092 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.612077    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.614592 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.614568    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-c05c9f-dockercfg-npplh\""
Sep 08 17:08:29.614719 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.614633    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-c05c9f\""
Sep 08 17:08:29.620433 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.620407    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:08:29.702733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.702699    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:08:29.706166 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.706147    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:29.723527 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.723498    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:08:29.797830 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.797791    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.798379 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.798350    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfwk\" (UniqueName: \"kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.900016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.899917    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.900016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.899967    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfwk\" (UniqueName: \"kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.900016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.900012    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:29.900232 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.900048    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5pv\" (UniqueName: \"kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:29.900688 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.900667    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.908038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.908007    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfwk\" (UniqueName: \"kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk\") pod \"payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:29.922832 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:29.922810    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:30.000775 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.000743    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.000927 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.000793    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5pv\" (UniqueName: \"kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.001524 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.001504    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.009436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.009407    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5pv\" (UniqueName: \"kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv\") pod \"payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.016204 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.016183    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.192910 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.192873    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:08:30.195423 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:08:30.195394    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd319fe_5c42_46c0_9b6c_037abac0f8a6.slice/crio-a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7 WatchSource:0}: Error finding container a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7: Status 404 returned error can't find the container with id a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7
Sep 08 17:08:30.293850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.293822    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:08:30.296086 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:08:30.296062    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903344ac_7a1a_44e1_901d_4eeec854c8cf.slice/crio-d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5 WatchSource:0}: Error finding container d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5: Status 404 returned error can't find the container with id d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5
Sep 08 17:08:30.636714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.636605    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" event={"ID":"903344ac-7a1a-44e1-901d-4eeec854c8cf","Type":"ContainerStarted","Data":"ee1f9981c8c376ef3cc7b8b30827876635b0f9b13db1b6cee64e3ac5cf9dec29"}
Sep 08 17:08:30.636714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.636665    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" event={"ID":"903344ac-7a1a-44e1-901d-4eeec854c8cf","Type":"ContainerStarted","Data":"d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5"}
Sep 08 17:08:30.637199 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.636780    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:30.638423 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.638395    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" event={"ID":"0bd319fe-5c42-46c0-9b6c-037abac0f8a6","Type":"ContainerStarted","Data":"0e01010820413ed7578f35c23cf0d2aacd42a72b695a656aa30c9f5be65dbc11"}
Sep 08 17:08:30.638555 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.638448    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" event={"ID":"0bd319fe-5c42-46c0-9b6c-037abac0f8a6","Type":"ContainerStarted","Data":"a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7"}
Sep 08 17:08:30.638555 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.638478    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:30.666564 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.666516    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" podStartSLOduration=1.666502322 podStartE2EDuration="1.666502322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:30.665158341 +0000 UTC m=+1054.132364539" watchObservedRunningTime="2026-09-08 17:08:30.666502322 +0000 UTC m=+1054.133708511"
Sep 08 17:08:30.682900 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:30.682857    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" podStartSLOduration=1.682844598 podStartE2EDuration="1.682844598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:30.68141797 +0000 UTC m=+1054.148624164" watchObservedRunningTime="2026-09-08 17:08:30.682844598 +0000 UTC m=+1054.150050792"
Sep 08 17:08:41.645115 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:41.645082    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:08:41.645585 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:41.645163    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:08:58.597240 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.597195    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:08:58.602218 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.602197    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.604897 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.604874    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-338600\""
Sep 08 17:08:58.629746 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.629715    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:08:58.717140 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.717109    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:08:58.720399 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.720382    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.726447 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.726423    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-338600-dockercfg-x7wz8\""
Sep 08 17:08:58.740467 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.740439    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.740467 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.740468    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr2b\" (UniqueName: \"kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.755777 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.755748    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:08:58.841522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.841471    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.841522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.841516    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr2b\" (UniqueName: \"kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.841790 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.841592    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkq2h\" (UniqueName: \"kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.841790 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.841661    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.842343 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.842304    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.861889 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.861823    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr2b\" (UniqueName: \"kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b\") pod \"payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.914697 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.914657    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:58.942552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.942518    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.942689 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.942609    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkq2h\" (UniqueName: \"kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.943192 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.943174    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:58.958413 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:58.958379    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkq2h\" (UniqueName: \"kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h\") pod \"payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:59.030563 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.030528    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:59.204770 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.204737    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:08:59.207836 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:08:59.207807    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35e73bb_2be3_48ab_a4e4_27b908b237e9.slice/crio-ce265b5704b394d1e25ff1140f0a7e69ac8a68a64c905ef83f20f49c33fed92c WatchSource:0}: Error finding container ce265b5704b394d1e25ff1140f0a7e69ac8a68a64c905ef83f20f49c33fed92c: Status 404 returned error can't find the container with id ce265b5704b394d1e25ff1140f0a7e69ac8a68a64c905ef83f20f49c33fed92c
Sep 08 17:08:59.310622 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.310592    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:08:59.312620 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:08:59.312591    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6267107c_8b73_49c3_93ca_6cd1da0a5831.slice/crio-0c285e8757c13846c10d661c5fffe01eeecf7bb01c700beab685ccf95958013a WatchSource:0}: Error finding container 0c285e8757c13846c10d661c5fffe01eeecf7bb01c700beab685ccf95958013a: Status 404 returned error can't find the container with id 0c285e8757c13846c10d661c5fffe01eeecf7bb01c700beab685ccf95958013a
Sep 08 17:08:59.776717 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.776671    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" event={"ID":"6267107c-8b73-49c3-93ca-6cd1da0a5831","Type":"ContainerStarted","Data":"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"}
Sep 08 17:08:59.777142 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.776768    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" event={"ID":"6267107c-8b73-49c3-93ca-6cd1da0a5831","Type":"ContainerStarted","Data":"0c285e8757c13846c10d661c5fffe01eeecf7bb01c700beab685ccf95958013a"}
Sep 08 17:08:59.777142 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.776797    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:08:59.778138 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.778117    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" event={"ID":"a35e73bb-2be3-48ab-a4e4-27b908b237e9","Type":"ContainerStarted","Data":"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"}
Sep 08 17:08:59.778220 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.778152    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" event={"ID":"a35e73bb-2be3-48ab-a4e4-27b908b237e9","Type":"ContainerStarted","Data":"ce265b5704b394d1e25ff1140f0a7e69ac8a68a64c905ef83f20f49c33fed92c"}
Sep 08 17:08:59.778220 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.778182    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:08:59.800631 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.800588    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" podStartSLOduration=1.800574862 podStartE2EDuration="1.800574862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:59.798863405 +0000 UTC m=+1083.266069601" watchObservedRunningTime="2026-09-08 17:08:59.800574862 +0000 UTC m=+1083.267781097"
Sep 08 17:08:59.828522 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:08:59.828470    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" podStartSLOduration=1.828454085 podStartE2EDuration="1.828454085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:08:59.827739869 +0000 UTC m=+1083.294946064" watchObservedRunningTime="2026-09-08 17:08:59.828454085 +0000 UTC m=+1083.295660278"
Sep 08 17:09:04.186356 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.186308    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:09:04.189859 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.189839    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.194043 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.194022    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d7359b-dockercfg-z7q7x\""
Sep 08 17:09:04.194384 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.194367    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d7359b\""
Sep 08 17:09:04.200980 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.200957    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:09:04.274766 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.274734    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:09:04.278036 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.278019    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.288493 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.288472    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjlj\" (UniqueName: \"kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.288593 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.288500    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.294999 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.294973    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:09:04.389885 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.389848    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwv7w\" (UniqueName: \"kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.390060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.389891    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.390060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.390009    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjlj\" (UniqueName: \"kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.390060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.390039    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.390706 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.390688    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.400077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.400054    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjlj\" (UniqueName: \"kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj\") pod \"payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.490721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.490628    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwv7w\" (UniqueName: \"kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.490721 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.490673    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.491305 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.491279    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.499861 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.499838    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:04.500901 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.500871    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwv7w\" (UniqueName: \"kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w\") pod \"payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.587692 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.587653    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:04.780299 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.780272    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:09:04.782787 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:09:04.782757    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb5a97_c70e_4fc9_a2b1_aa7a981dc4fb.slice/crio-e497bcf3323231b592d0c93b9852cb433bb8a9ac460295be25fb3474868340de WatchSource:0}: Error finding container e497bcf3323231b592d0c93b9852cb433bb8a9ac460295be25fb3474868340de: Status 404 returned error can't find the container with id e497bcf3323231b592d0c93b9852cb433bb8a9ac460295be25fb3474868340de
Sep 08 17:09:04.803062 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.803032    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" event={"ID":"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb","Type":"ContainerStarted","Data":"e497bcf3323231b592d0c93b9852cb433bb8a9ac460295be25fb3474868340de"}
Sep 08 17:09:04.863517 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:04.863491    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:09:04.865954 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:09:04.865926    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda065e421_40b6_4fcb_b1b0_42423cf5abd2.slice/crio-b8e6261b3efa201d54a06fe41dde8386472097159acfe2b08c239d7d7b57c010 WatchSource:0}: Error finding container b8e6261b3efa201d54a06fe41dde8386472097159acfe2b08c239d7d7b57c010: Status 404 returned error can't find the container with id b8e6261b3efa201d54a06fe41dde8386472097159acfe2b08c239d7d7b57c010
Sep 08 17:09:05.809125 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.809079    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" event={"ID":"a065e421-40b6-4fcb-b1b0-42423cf5abd2","Type":"ContainerStarted","Data":"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"}
Sep 08 17:09:05.809575 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.809254    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:05.809575 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.809297    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" event={"ID":"a065e421-40b6-4fcb-b1b0-42423cf5abd2","Type":"ContainerStarted","Data":"b8e6261b3efa201d54a06fe41dde8386472097159acfe2b08c239d7d7b57c010"}
Sep 08 17:09:05.810684 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.810664    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" event={"ID":"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb","Type":"ContainerStarted","Data":"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"}
Sep 08 17:09:05.810764 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.810722    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:05.828487 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.828442    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" podStartSLOduration=1.828424773 podStartE2EDuration="1.828424773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:05.827552796 +0000 UTC m=+1089.294758990" watchObservedRunningTime="2026-09-08 17:09:05.828424773 +0000 UTC m=+1089.295630967"
Sep 08 17:09:05.855023 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:05.854974    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" podStartSLOduration=1.854958128 podStartE2EDuration="1.854958128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:05.85079912 +0000 UTC m=+1089.318005313" watchObservedRunningTime="2026-09-08 17:09:05.854958128 +0000 UTC m=+1089.322164320"
Sep 08 17:09:10.785411 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:10.785376    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:09:10.785832 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:10.785466    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:09:16.816829 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:16.816798    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:09:16.817312 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:16.817017    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:09:25.622592 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.622556    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:09:25.625783 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.625765    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.631569 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.631546    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"auth-test-model-c05c9f-kserve-self-signed-certs\""
Sep 08 17:09:25.632616 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.632593    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"default-dockercfg-mb7df\""
Sep 08 17:09:25.632741 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.632721    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"openshift-service-ca.crt\""
Sep 08 17:09:25.632816 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.632738    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"kube-root-ca.crt\""
Sep 08 17:09:25.646610 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.646579    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:09:25.681087 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681056    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.681239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681096    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.681239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681125    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvt8\" (UniqueName: \"kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.681239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681178    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.681239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681222    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.681425 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.681344    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.781880 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781842    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781893    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781914    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781934    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvt8\" (UniqueName: \"kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781959    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782064 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.781988    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.782294    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.782349    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.782409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.782388    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.784222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.784198    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.784527 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.784509    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.792103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.792080    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvt8\" (UniqueName: \"kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8\") pod \"auth-test-model-c05c9f-kserve-cf87f4687-krrn9\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:25.935197 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:25.935161    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:26.208883 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:26.208861    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:09:26.211133 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:09:26.211104    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6d10b18_4f26_4e2a_abe5_16667900cded.slice/crio-180d2b51151fc98c949b011eaa394d1c9c25c3c373fbb42f443f916c9fcb7965 WatchSource:0}: Error finding container 180d2b51151fc98c949b011eaa394d1c9c25c3c373fbb42f443f916c9fcb7965: Status 404 returned error can't find the container with id 180d2b51151fc98c949b011eaa394d1c9c25c3c373fbb42f443f916c9fcb7965
Sep 08 17:09:26.912005 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:26.911967    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" event={"ID":"f6d10b18-4f26-4e2a-abe5-16667900cded","Type":"ContainerStarted","Data":"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"}
Sep 08 17:09:26.912388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:26.912015    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" event={"ID":"f6d10b18-4f26-4e2a-abe5-16667900cded","Type":"ContainerStarted","Data":"180d2b51151fc98c949b011eaa394d1c9c25c3c373fbb42f443f916c9fcb7965"}
Sep 08 17:09:26.935148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:26.935100    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" podStartSLOduration=1.9350859649999999 podStartE2EDuration="1.935085965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:26.934532833 +0000 UTC m=+1110.401739025" watchObservedRunningTime="2026-09-08 17:09:26.935085965 +0000 UTC m=+1110.402292188"
Sep 08 17:09:35.935478 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:35.935436    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:35.935980 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:35.935493    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:35.948275 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:35.948244    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:35.963714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:35.963688    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:09:42.642827 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.642791    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:09:42.647473 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.647447    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.652814 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.652796    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb1565be\""
Sep 08 17:09:42.653524 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.653506    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb1565be-dockercfg-fbq74\""
Sep 08 17:09:42.678641 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.678612    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:09:42.727696 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.727660    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.727858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.727783    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zmt\" (UniqueName: \"kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.828222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.828185    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.828425 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.828254    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zmt\" (UniqueName: \"kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.828912 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.828890    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.837363 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.837343    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zmt\" (UniqueName: \"kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt\") pod \"payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") " pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:42.957928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:42.957897    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:43.255108 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:43.255021    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:09:43.258849 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:09:43.258824    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79d134d_a5d4_441b_b77d_55c861497e48.slice/crio-25006a17bf3c8aa3d08ff4a4e14b7868f16ef7d506c35100600869d2dab8e1ee WatchSource:0}: Error finding container 25006a17bf3c8aa3d08ff4a4e14b7868f16ef7d506c35100600869d2dab8e1ee: Status 404 returned error can't find the container with id 25006a17bf3c8aa3d08ff4a4e14b7868f16ef7d506c35100600869d2dab8e1ee
Sep 08 17:09:43.994056 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:43.994024    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" event={"ID":"d79d134d-a5d4-441b-b77d-55c861497e48","Type":"ContainerStarted","Data":"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"}
Sep 08 17:09:43.994482 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:43.994080    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" event={"ID":"d79d134d-a5d4-441b-b77d-55c861497e48","Type":"ContainerStarted","Data":"25006a17bf3c8aa3d08ff4a4e14b7868f16ef7d506c35100600869d2dab8e1ee"}
Sep 08 17:09:43.994482 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:43.994106    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:09:44.014076 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:44.014021    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" podStartSLOduration=2.014006483 podStartE2EDuration="2.014006483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:09:44.011319666 +0000 UTC m=+1127.478525859" watchObservedRunningTime="2026-09-08 17:09:44.014006483 +0000 UTC m=+1127.481212689"
Sep 08 17:09:55.001001 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:09:55.000968    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:10:01.524196 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.524158    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:10:01.527623 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.527605    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.530773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.530751    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"auth-test-model-338600-kserve-self-signed-certs\""
Sep 08 17:10:01.531748 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.531732    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"openshift-service-ca.crt\""
Sep 08 17:10:01.531868 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.531850    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"kube-root-ca.crt\""
Sep 08 17:10:01.531993 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.531980    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"default-dockercfg-4wmmt\""
Sep 08 17:10:01.539296 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.539274    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:10:01.594273 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594241    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.594441 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594277    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhg6\" (UniqueName: \"kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.594441 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594299    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.594441 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594382    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.594441 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594428    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.594605 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.594444    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.695964 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.695932    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696145 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.695976    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhg6\" (UniqueName: \"kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696145 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696097    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696145 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696138    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696319 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696213    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696319 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696242    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696471 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696446    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696532 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696501    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.696591 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.696538    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.698503 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.698478    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.698858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.698839    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.710647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.710621    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhg6\" (UniqueName: \"kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6\") pod \"auth-test-model-338600-kserve-86b768b967-2llp2\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") " pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:01.837351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:01.837251    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:02.108885 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:02.108807    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:10:02.113479 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:10:02.113447    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc69e9466_c03e_4b35_84bc_c4a812b54080.slice/crio-8dfb9a423e5898731d2efe261b397ca18e6a3fd2fbf26fa751b3b321639504fb WatchSource:0}: Error finding container 8dfb9a423e5898731d2efe261b397ca18e6a3fd2fbf26fa751b3b321639504fb: Status 404 returned error can't find the container with id 8dfb9a423e5898731d2efe261b397ca18e6a3fd2fbf26fa751b3b321639504fb
Sep 08 17:10:03.084167 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:03.084126    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" event={"ID":"c69e9466-c03e-4b35-84bc-c4a812b54080","Type":"ContainerStarted","Data":"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"}
Sep 08 17:10:03.084827 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:03.084176    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" event={"ID":"c69e9466-c03e-4b35-84bc-c4a812b54080","Type":"ContainerStarted","Data":"8dfb9a423e5898731d2efe261b397ca18e6a3fd2fbf26fa751b3b321639504fb"}
Sep 08 17:10:03.103280 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:03.103225    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" podStartSLOduration=2.103207156 podStartE2EDuration="2.103207156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:10:03.101597568 +0000 UTC m=+1146.568803763" watchObservedRunningTime="2026-09-08 17:10:03.103207156 +0000 UTC m=+1146.570413348"
Sep 08 17:10:11.838241 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:11.838200    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:11.838713 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:11.838274    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:11.851410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:11.851383    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:12.137037 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:12.136955    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:10:23.312341 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.312254    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:10:23.315712 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.315690    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.318465 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.318444    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb1565be\"/\"openshift-service-ca.crt\""
Sep 08 17:10:23.319566 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.319544    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb1565be\"/\"default-dockercfg-tj9hq\""
Sep 08 17:10:23.319665 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.319585    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-cb1565be\"/\"e2e-lifecycle-model-cb1565be-kserve-self-signed-certs\""
Sep 08 17:10:23.319665 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.319611    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-cb1565be\"/\"kube-root-ca.crt\""
Sep 08 17:10:23.325148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.325126    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:10:23.389700 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389660    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.389859 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389716    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rffzv\" (UniqueName: \"kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.389859 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389760    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.389859 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389785    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.389859 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389837    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.390001 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.389869    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490598 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490562    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490608    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490642    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490670    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rffzv\" (UniqueName: \"kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490688    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.490780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.490703    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.491072 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.491036    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.491072 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.491048    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.491176 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.491112    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.492902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.492881    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.493158 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.493138    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.501024 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.501002    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rffzv\" (UniqueName: \"kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv\") pod \"e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") " pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.626001 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.625899    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:23.896194 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:23.896168    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:10:23.897978 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:10:23.897948    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode909dafa_aef2_4c19_b14b_c7ec8fbd6152.slice/crio-b4489faeaacb9495ba8c0887699025bae5cfba2e174fd8b50519c9870b85d416 WatchSource:0}: Error finding container b4489faeaacb9495ba8c0887699025bae5cfba2e174fd8b50519c9870b85d416: Status 404 returned error can't find the container with id b4489faeaacb9495ba8c0887699025bae5cfba2e174fd8b50519c9870b85d416
Sep 08 17:10:24.184455 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:24.184416    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" event={"ID":"e909dafa-aef2-4c19-b14b-c7ec8fbd6152","Type":"ContainerStarted","Data":"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"}
Sep 08 17:10:24.184455 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:24.184462    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" event={"ID":"e909dafa-aef2-4c19-b14b-c7ec8fbd6152","Type":"ContainerStarted","Data":"b4489faeaacb9495ba8c0887699025bae5cfba2e174fd8b50519c9870b85d416"}
Sep 08 17:10:24.203407 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:24.203358    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" podStartSLOduration=1.203315648 podStartE2EDuration="1.203315648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:10:24.201948185 +0000 UTC m=+1167.669154379" watchObservedRunningTime="2026-09-08 17:10:24.203315648 +0000 UTC m=+1167.670521840"
Sep 08 17:10:33.626660 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:33.626617    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:33.627137 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:33.626673    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:33.639210 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:33.639183    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:34.249230 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:34.249200    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:10:57.022715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:57.022684    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:10:57.024046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:10:57.024025    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:11:05.466199 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.466162    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:11:05.470255 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.466441    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" podUID="d79d134d-a5d4-441b-b77d-55c861497e48" containerName="payload-processing" containerID="cri-o://8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687" gracePeriod=30
Sep 08 17:11:05.718878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.718814    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:11:05.783090 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.783052    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume\") pod \"d79d134d-a5d4-441b-b77d-55c861497e48\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") "
Sep 08 17:11:05.783282 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.783111    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9zmt\" (UniqueName: \"kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt\") pod \"d79d134d-a5d4-441b-b77d-55c861497e48\" (UID: \"d79d134d-a5d4-441b-b77d-55c861497e48\") "
Sep 08 17:11:05.783454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.783424    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d79d134d-a5d4-441b-b77d-55c861497e48" (UID: "d79d134d-a5d4-441b-b77d-55c861497e48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:11:05.785160 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.785135    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt" (OuterVolumeSpecName: "kube-api-access-f9zmt") pod "d79d134d-a5d4-441b-b77d-55c861497e48" (UID: "d79d134d-a5d4-441b-b77d-55c861497e48"). InnerVolumeSpecName "kube-api-access-f9zmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:11:05.883929 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.883888    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zmt\" (UniqueName: \"kubernetes.io/projected/d79d134d-a5d4-441b-b77d-55c861497e48-kube-api-access-f9zmt\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:05.883929 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:05.883922    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d79d134d-a5d4-441b-b77d-55c861497e48-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:06.397577 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.397542    2542 generic.go:356] "Generic (PLEG): container finished" podID="d79d134d-a5d4-441b-b77d-55c861497e48" containerID="8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687" exitCode=0
Sep 08 17:11:06.397787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.397585    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"
Sep 08 17:11:06.397787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.397623    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" event={"ID":"d79d134d-a5d4-441b-b77d-55c861497e48","Type":"ContainerDied","Data":"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"}
Sep 08 17:11:06.397787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.397672    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc" event={"ID":"d79d134d-a5d4-441b-b77d-55c861497e48","Type":"ContainerDied","Data":"25006a17bf3c8aa3d08ff4a4e14b7868f16ef7d506c35100600869d2dab8e1ee"}
Sep 08 17:11:06.397787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.397694    2542 scope.go:117] "RemoveContainer" containerID="8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"
Sep 08 17:11:06.406765 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.406734    2542 scope.go:117] "RemoveContainer" containerID="8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"
Sep 08 17:11:06.407025 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:11:06.407004    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687\": container with ID starting with 8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687 not found: ID does not exist" containerID="8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"
Sep 08 17:11:06.407095 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.407033    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687"} err="failed to get container status \"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687\": rpc error: code = NotFound desc = could not find container \"8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687\": container with ID starting with 8fbb87cc5dc56bbf4262e8ba7bb363499205d9e65bda12d987b72e84862e7687 not found: ID does not exist"
Sep 08 17:11:06.418559 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.418537    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:11:06.422375 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:06.422351    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb1565be-7d79d6c877-z5svc"]
Sep 08 17:11:07.123174 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:07.123139    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d79d134d-a5d4-441b-b77d-55c861497e48" path="/var/lib/kubelet/pods/d79d134d-a5d4-441b-b77d-55c861497e48/volumes"
Sep 08 17:11:13.647211 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:13.647173    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:11:28.666812 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.666761    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" podUID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" containerName="main" containerID="cri-o://a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0" gracePeriod=45
Sep 08 17:11:28.917202 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.917136    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:11:28.997448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997410    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997453    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rffzv\" (UniqueName: \"kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997476    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997541    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997577    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997635    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home\") pod \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\" (UID: \"e909dafa-aef2-4c19-b14b-c7ec8fbd6152\") "
Sep 08 17:11:28.997856 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997823    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:11:28.997856 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997845    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache" (OuterVolumeSpecName: "model-cache") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:11:28.997976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997931    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:28.998035 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.997997    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home" (OuterVolumeSpecName: "home") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:11:28.999660 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.999630    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm" (OuterVolumeSpecName: "dshm") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:11:28.999778 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.999704    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv" (OuterVolumeSpecName: "kube-api-access-rffzv") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "kube-api-access-rffzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:11:28.999778 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:28.999742    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e909dafa-aef2-4c19-b14b-c7ec8fbd6152" (UID: "e909dafa-aef2-4c19-b14b-c7ec8fbd6152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:11:29.099103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.099060    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:29.099103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.099093    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:29.099103 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.099103    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rffzv\" (UniqueName: \"kubernetes.io/projected/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-kube-api-access-rffzv\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:29.099360 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.099115    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:29.099360 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.099124    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e909dafa-aef2-4c19-b14b-c7ec8fbd6152-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:11:29.511479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.511442    2542 generic.go:356] "Generic (PLEG): container finished" podID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" containerID="a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0" exitCode=0
Sep 08 17:11:29.511479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.511482    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" event={"ID":"e909dafa-aef2-4c19-b14b-c7ec8fbd6152","Type":"ContainerDied","Data":"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"}
Sep 08 17:11:29.511708 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.511516    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c" event={"ID":"e909dafa-aef2-4c19-b14b-c7ec8fbd6152","Type":"ContainerDied","Data":"b4489faeaacb9495ba8c0887699025bae5cfba2e174fd8b50519c9870b85d416"}
Sep 08 17:11:29.511708 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.511533    2542 scope.go:117] "RemoveContainer" containerID="a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"
Sep 08 17:11:29.511708 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.511484    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"
Sep 08 17:11:29.522594 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.522564    2542 scope.go:117] "RemoveContainer" containerID="a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"
Sep 08 17:11:29.522979 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:11:29.522947    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0\": container with ID starting with a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0 not found: ID does not exist" containerID="a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"
Sep 08 17:11:29.522979 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.522975    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0"} err="failed to get container status \"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0\": rpc error: code = NotFound desc = could not find container \"a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0\": container with ID starting with a8ac7268afdae522d840737d9adf550a34f3ca1c5c73805486dbc04d2ccecba0 not found: ID does not exist"
Sep 08 17:11:29.529472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.529448    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:11:29.538294 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:29.538265    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-cb1565be/e2e-lifecycle-model-cb1565be-kserve-6469fb9f79-c6b6c"]
Sep 08 17:11:31.118388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:31.118355    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" path="/var/lib/kubelet/pods/e909dafa-aef2-4c19-b14b-c7ec8fbd6152/volumes"
Sep 08 17:11:57.088237 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:57.088160    2542 scope.go:117] "RemoveContainer" containerID="f8c482f8687dad583d3713314906a15420e6be43cc5593680ae275e92054623d"
Sep 08 17:11:57.096540 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:11:57.096521    2542 scope.go:117] "RemoveContainer" containerID="2f65976cee4c3a613c304515d3ca13b859ced68b42350d1dbae7c90001574672"
Sep 08 17:12:34.149757 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:34.149719    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:12:35.284453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:35.284416    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:12:49.169319 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.169273    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" podUID="f6d10b18-4f26-4e2a-abe5-16667900cded" containerName="main" containerID="cri-o://8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27" gracePeriod=45
Sep 08 17:12:49.415550 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.415524    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:12:49.539407 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539374    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539422    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539459    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mvt8\" (UniqueName: \"kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539514    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539537    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539558    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home\") pod \"f6d10b18-4f26-4e2a-abe5-16667900cded\" (UID: \"f6d10b18-4f26-4e2a-abe5-16667900cded\") "
Sep 08 17:12:49.539878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539708    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:49.539878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539804    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:49.539878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539824    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home" (OuterVolumeSpecName: "home") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:49.539878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.539846    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.541864 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.541839    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm" (OuterVolumeSpecName: "dshm") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:49.541864 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.541853    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8" (OuterVolumeSpecName: "kube-api-access-8mvt8") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "kube-api-access-8mvt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:12:49.541864 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.541861    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6d10b18-4f26-4e2a-abe5-16667900cded" (UID: "f6d10b18-4f26-4e2a-abe5-16667900cded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:12:49.641156 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.641117    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvt8\" (UniqueName: \"kubernetes.io/projected/f6d10b18-4f26-4e2a-abe5-16667900cded-kube-api-access-8mvt8\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.641156 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.641150    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.641156 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.641160    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.641430 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.641169    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6d10b18-4f26-4e2a-abe5-16667900cded-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.641430 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.641177    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d10b18-4f26-4e2a-abe5-16667900cded-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:49.897630 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.897538    2542 generic.go:356] "Generic (PLEG): container finished" podID="f6d10b18-4f26-4e2a-abe5-16667900cded" containerID="8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27" exitCode=0
Sep 08 17:12:49.897630 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.897597    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"
Sep 08 17:12:49.897807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.897631    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" event={"ID":"f6d10b18-4f26-4e2a-abe5-16667900cded","Type":"ContainerDied","Data":"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"}
Sep 08 17:12:49.897807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.897679    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9" event={"ID":"f6d10b18-4f26-4e2a-abe5-16667900cded","Type":"ContainerDied","Data":"180d2b51151fc98c949b011eaa394d1c9c25c3c373fbb42f443f916c9fcb7965"}
Sep 08 17:12:49.897807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.897698    2542 scope.go:117] "RemoveContainer" containerID="8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"
Sep 08 17:12:49.906990 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.906970    2542 scope.go:117] "RemoveContainer" containerID="8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"
Sep 08 17:12:49.907270 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:12:49.907251    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27\": container with ID starting with 8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27 not found: ID does not exist" containerID="8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"
Sep 08 17:12:49.907357 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.907284    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27"} err="failed to get container status \"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27\": rpc error: code = NotFound desc = could not find container \"8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27\": container with ID starting with 8e7502df3a4566957683ee7f1a66262d042f6d1b36f9cd882cf9838cfd69da27 not found: ID does not exist"
Sep 08 17:12:49.917716 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.917689    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:12:49.922231 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:49.922211    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/auth-test-model-c05c9f-kserve-cf87f4687-krrn9"]
Sep 08 17:12:50.851171 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:50.851124    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" podUID="c69e9466-c03e-4b35-84bc-c4a812b54080" containerName="main" containerID="cri-o://ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491" gracePeriod=45
Sep 08 17:12:51.103864 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.103797    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:12:51.116676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.116649    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6d10b18-4f26-4e2a-abe5-16667900cded" path="/var/lib/kubelet/pods/f6d10b18-4f26-4e2a-abe5-16667900cded/volumes"
Sep 08 17:12:51.155445 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155410    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155634 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155477    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155634 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155500    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlhg6\" (UniqueName: \"kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155634 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155552    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155634 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155567    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155634 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155623    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm\") pod \"c69e9466-c03e-4b35-84bc-c4a812b54080\" (UID: \"c69e9466-c03e-4b35-84bc-c4a812b54080\") "
Sep 08 17:12:51.155932 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155746    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache" (OuterVolumeSpecName: "model-cache") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:51.155932 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155830    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:51.155932 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155880    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home" (OuterVolumeSpecName: "home") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:51.155932 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.155887    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.157857 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.157826    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6" (OuterVolumeSpecName: "kube-api-access-wlhg6") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "kube-api-access-wlhg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:12:51.157857 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.157835    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm" (OuterVolumeSpecName: "dshm") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:12:51.158010 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.157838    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c69e9466-c03e-4b35-84bc-c4a812b54080" (UID: "c69e9466-c03e-4b35-84bc-c4a812b54080"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:12:51.256571 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.256537    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.256571 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.256573    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhg6\" (UniqueName: \"kubernetes.io/projected/c69e9466-c03e-4b35-84bc-c4a812b54080-kube-api-access-wlhg6\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.256761 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.256585    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.256761 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.256594    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c69e9466-c03e-4b35-84bc-c4a812b54080-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.256761 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.256603    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c69e9466-c03e-4b35-84bc-c4a812b54080-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:12:51.910219 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.910182    2542 generic.go:356] "Generic (PLEG): container finished" podID="c69e9466-c03e-4b35-84bc-c4a812b54080" containerID="ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491" exitCode=0
Sep 08 17:12:51.910733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.910225    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"
Sep 08 17:12:51.910733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.910271    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" event={"ID":"c69e9466-c03e-4b35-84bc-c4a812b54080","Type":"ContainerDied","Data":"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"}
Sep 08 17:12:51.910733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.910316    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2" event={"ID":"c69e9466-c03e-4b35-84bc-c4a812b54080","Type":"ContainerDied","Data":"8dfb9a423e5898731d2efe261b397ca18e6a3fd2fbf26fa751b3b321639504fb"}
Sep 08 17:12:51.910733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.910361    2542 scope.go:117] "RemoveContainer" containerID="ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"
Sep 08 17:12:51.919041 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.919023    2542 scope.go:117] "RemoveContainer" containerID="ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"
Sep 08 17:12:51.919293 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:12:51.919271    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491\": container with ID starting with ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491 not found: ID does not exist" containerID="ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"
Sep 08 17:12:51.919371 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.919300    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491"} err="failed to get container status \"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491\": rpc error: code = NotFound desc = could not find container \"ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491\": container with ID starting with ac233afb79161d917f2baf925ead9d412461bf618ed3d8ff8028452af63f6491 not found: ID does not exist"
Sep 08 17:12:51.931428 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.931400    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:12:51.936720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:51.936698    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/auth-test-model-338600-kserve-86b768b967-2llp2"]
Sep 08 17:12:53.116870 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:53.116834    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69e9466-c03e-4b35-84bc-c4a812b54080" path="/var/lib/kubelet/pods/c69e9466-c03e-4b35-84bc-c4a812b54080/volumes"
Sep 08 17:12:55.184454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184419    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184760    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6d10b18-4f26-4e2a-abe5-16667900cded" containerName="main"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184772    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6d10b18-4f26-4e2a-abe5-16667900cded" containerName="main"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184782    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d79d134d-a5d4-441b-b77d-55c861497e48" containerName="payload-processing"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184788    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="d79d134d-a5d4-441b-b77d-55c861497e48" containerName="payload-processing"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184798    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" containerName="main"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184805    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" containerName="main"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184821    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69e9466-c03e-4b35-84bc-c4a812b54080" containerName="main"
Sep 08 17:12:55.184850 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184825    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69e9466-c03e-4b35-84bc-c4a812b54080" containerName="main"
Sep 08 17:12:55.185096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184874    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="e909dafa-aef2-4c19-b14b-c7ec8fbd6152" containerName="main"
Sep 08 17:12:55.185096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184884    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69e9466-c03e-4b35-84bc-c4a812b54080" containerName="main"
Sep 08 17:12:55.185096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184890    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6d10b18-4f26-4e2a-abe5-16667900cded" containerName="main"
Sep 08 17:12:55.185096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.184895    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="d79d134d-a5d4-441b-b77d-55c861497e48" containerName="payload-processing"
Sep 08 17:12:55.189267 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.189249    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.191854 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.191833    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f0d25d\""
Sep 08 17:12:55.192132 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.192114    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f0d25d-dockercfg-rkmgh\""
Sep 08 17:12:55.202127 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.202104    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:12:55.204364 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.204342    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:12:55.207598 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.207582    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.221099 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.221075    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:12:55.296217 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.296183    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.296410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.296228    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxgvt\" (UniqueName: \"kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.296410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.296284    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.296410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.296393    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8stk\" (UniqueName: \"kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.398109 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.398036    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8stk\" (UniqueName: \"kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.398302 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.398179    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.398302 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.398221    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxgvt\" (UniqueName: \"kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.398302 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.398238    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.399095 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.399073    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.399287 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.399244    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.407764 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.407734    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxgvt\" (UniqueName: \"kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt\") pod \"payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.409081 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.409062    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8stk\" (UniqueName: \"kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk\") pod \"payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") " pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.499635 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.499535    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.516808 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.516768    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.770249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.770223    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:12:55.772164 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:12:55.772138    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a52e54a_3964_4d6e_a991_9f301bd10f58.slice/crio-aa9e800bafdf7b45e660c7e4fc5c72ff139f1b9fba8cb09ceda32545be813ef3 WatchSource:0}: Error finding container aa9e800bafdf7b45e660c7e4fc5c72ff139f1b9fba8cb09ceda32545be813ef3: Status 404 returned error can't find the container with id aa9e800bafdf7b45e660c7e4fc5c72ff139f1b9fba8cb09ceda32545be813ef3
Sep 08 17:12:55.803004 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.802975    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:12:55.812517 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:12:55.812478    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55b5f98_0962_4090_b42a_b3cb07980875.slice/crio-0f36733cca30ea70b5f41a7e999d4f98e1432f65da84c428812c1a114d91f2a6 WatchSource:0}: Error finding container 0f36733cca30ea70b5f41a7e999d4f98e1432f65da84c428812c1a114d91f2a6: Status 404 returned error can't find the container with id 0f36733cca30ea70b5f41a7e999d4f98e1432f65da84c428812c1a114d91f2a6
Sep 08 17:12:55.931745 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.931698    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" event={"ID":"a55b5f98-0962-4090-b42a-b3cb07980875","Type":"ContainerStarted","Data":"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"}
Sep 08 17:12:55.931938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.931769    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" event={"ID":"a55b5f98-0962-4090-b42a-b3cb07980875","Type":"ContainerStarted","Data":"0f36733cca30ea70b5f41a7e999d4f98e1432f65da84c428812c1a114d91f2a6"}
Sep 08 17:12:55.931938 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.931838    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:12:55.933819 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.933794    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" event={"ID":"9a52e54a-3964-4d6e-a991-9f301bd10f58","Type":"ContainerStarted","Data":"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"}
Sep 08 17:12:55.933942 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.933840    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" event={"ID":"9a52e54a-3964-4d6e-a991-9f301bd10f58","Type":"ContainerStarted","Data":"aa9e800bafdf7b45e660c7e4fc5c72ff139f1b9fba8cb09ceda32545be813ef3"}
Sep 08 17:12:55.933942 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.933862    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:12:55.948683 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.948620    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" podStartSLOduration=0.948601763 podStartE2EDuration="948.601763ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:55.947382345 +0000 UTC m=+1319.414588538" watchObservedRunningTime="2026-09-08 17:12:55.948601763 +0000 UTC m=+1319.415807959"
Sep 08 17:12:55.963822 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:55.963773    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" podStartSLOduration=0.963754424 podStartE2EDuration="963.754424ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:55.962143325 +0000 UTC m=+1319.429349520" watchObservedRunningTime="2026-09-08 17:12:55.963754424 +0000 UTC m=+1319.430960618"
Sep 08 17:12:58.990949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:58.990913    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:12:58.995559 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:58.995543    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.003892 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.003870    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"default-dockercfg-mb7df\""
Sep 08 17:12:59.004888 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.004872    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"sub-test-model-c05c9f-kserve-self-signed-certs\""
Sep 08 17:12:59.004943 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.004904    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"kube-root-ca.crt\""
Sep 08 17:12:59.004943 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.004921    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"openshift-service-ca.crt\""
Sep 08 17:12:59.011717 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.011691    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:12:59.127539 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127510    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.127539 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127541    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.127762 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127566    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.127762 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127615    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.127762 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127704    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.127932 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.127731    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzftf\" (UniqueName: \"kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229532 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229489    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229540    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzftf\" (UniqueName: \"kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229608    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229634    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229666    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.229715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229693    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.230002 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229975    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.230063 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.229991    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.230116 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.230096    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.231944 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.231923    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.232196 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.232176    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.238130 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.238108    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzftf\" (UniqueName: \"kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf\") pod \"sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.304453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.304370    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:12:59.576605 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.576580    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:12:59.578759 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:12:59.578730    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b1846f_7d58_4a36_9eb2_caca9d29cdd7.slice/crio-67d1641b90dbdfbb8e407ed81bae722f11d4ca3fac450a58a70882e63fab0380 WatchSource:0}: Error finding container 67d1641b90dbdfbb8e407ed81bae722f11d4ca3fac450a58a70882e63fab0380: Status 404 returned error can't find the container with id 67d1641b90dbdfbb8e407ed81bae722f11d4ca3fac450a58a70882e63fab0380
Sep 08 17:12:59.954983 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.954951    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" event={"ID":"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7","Type":"ContainerStarted","Data":"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"}
Sep 08 17:12:59.955157 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.954990    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" event={"ID":"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7","Type":"ContainerStarted","Data":"67d1641b90dbdfbb8e407ed81bae722f11d4ca3fac450a58a70882e63fab0380"}
Sep 08 17:12:59.974964 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:12:59.974910    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" podStartSLOduration=1.9748946379999999 podStartE2EDuration="1.974894638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:12:59.973691975 +0000 UTC m=+1323.440898173" watchObservedRunningTime="2026-09-08 17:12:59.974894638 +0000 UTC m=+1323.442100830"
Sep 08 17:13:06.942311 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:06.942276    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:13:06.942840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:06.942378    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:13:09.305274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:09.305219    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:13:09.305274 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:09.305285    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:13:09.318277 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:09.318253    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:13:10.014976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:10.014944    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:13:24.380045 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.379964    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:13:24.383672 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.383650    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.387082 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.387063    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b9f2b8-dockercfg-xmqhc\""
Sep 08 17:13:24.387474 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.387460    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b9f2b8\""
Sep 08 17:13:24.399290 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.399260    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:13:24.466217 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.466185    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.466428 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.466231    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8cm\" (UniqueName: \"kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.567520 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.567466    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.567520 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.567514    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8cm\" (UniqueName: \"kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.568239 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.568219    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.576076 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.576049    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8cm\" (UniqueName: \"kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm\") pod \"payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.696523 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.696491    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:24.969732 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:24.969704    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:13:24.971621 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:13:24.971596    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ad4a2f_b719_4a21_ab77_1663a947f5cf.slice/crio-eb069e63f19e848c36d0a3e12b645d422b80eb61907cd5022838c98588972407 WatchSource:0}: Error finding container eb069e63f19e848c36d0a3e12b645d422b80eb61907cd5022838c98588972407: Status 404 returned error can't find the container with id eb069e63f19e848c36d0a3e12b645d422b80eb61907cd5022838c98588972407
Sep 08 17:13:25.076521 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:25.076489    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" event={"ID":"74ad4a2f-b719-4a21-ab77-1663a947f5cf","Type":"ContainerStarted","Data":"a92e0805076c12b872048271e6cfe18259ce06f8961250e994f91593cfd8cfbe"}
Sep 08 17:13:25.076702 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:25.076538    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" event={"ID":"74ad4a2f-b719-4a21-ab77-1663a947f5cf","Type":"ContainerStarted","Data":"eb069e63f19e848c36d0a3e12b645d422b80eb61907cd5022838c98588972407"}
Sep 08 17:13:25.076702 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:25.076598    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:25.092851 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:25.092790    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" podStartSLOduration=1.0927698 podStartE2EDuration="1.0927698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:13:25.092166502 +0000 UTC m=+1348.559372700" watchObservedRunningTime="2026-09-08 17:13:25.0927698 +0000 UTC m=+1348.559975993"
Sep 08 17:13:29.394685 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.394649    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:13:29.398792 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.398772    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.401550 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.401524    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"kube-root-ca.crt\""
Sep 08 17:13:29.402928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.402908    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"default-dockercfg-4wmmt\""
Sep 08 17:13:29.402928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.402921    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"openshift-service-ca.crt\""
Sep 08 17:13:29.403106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.402940    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"sub-test-model-338600-kserve-self-signed-certs\""
Sep 08 17:13:29.406962 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.406943    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:13:29.508457 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508412    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tr5q\" (UniqueName: \"kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.508639 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508466    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.508639 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508556    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.508639 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508591    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.508802 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508696    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.508802 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.508720    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610136 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610094    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tr5q\" (UniqueName: \"kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610136 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610138    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610173    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610209    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610279    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610300    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610632 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610595    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610677 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610606    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.610677 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.610664    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.612568 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.612540    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.612891 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.612872    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.624388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.624365    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tr5q\" (UniqueName: \"kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q\") pod \"sub-test-model-338600-kserve-675c7b9659-j5dwt\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") " pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.709473 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.709437    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:29.984949 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:29.984922    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:13:29.986757 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:13:29.986735    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680c5eae_8162_4d51_8141_6bc590e6443b.slice/crio-6ff74b32f483533ef25bc771fcc088c7334825ecab4b69d2c8b92db782ed5d5a WatchSource:0}: Error finding container 6ff74b32f483533ef25bc771fcc088c7334825ecab4b69d2c8b92db782ed5d5a: Status 404 returned error can't find the container with id 6ff74b32f483533ef25bc771fcc088c7334825ecab4b69d2c8b92db782ed5d5a
Sep 08 17:13:30.101022 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:30.100979    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" event={"ID":"680c5eae-8162-4d51-8141-6bc590e6443b","Type":"ContainerStarted","Data":"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"}
Sep 08 17:13:30.101022 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:30.101029    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" event={"ID":"680c5eae-8162-4d51-8141-6bc590e6443b","Type":"ContainerStarted","Data":"6ff74b32f483533ef25bc771fcc088c7334825ecab4b69d2c8b92db782ed5d5a"}
Sep 08 17:13:30.136368 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:30.136293    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" podStartSLOduration=1.136276758 podStartE2EDuration="1.136276758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:13:30.13535061 +0000 UTC m=+1353.602556796" watchObservedRunningTime="2026-09-08 17:13:30.136276758 +0000 UTC m=+1353.603482952"
Sep 08 17:13:36.082735 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:36.082701    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:13:39.709790 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:39.709749    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:39.710285 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:39.709831    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:39.722449 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:39.722423    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:13:40.161564 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:13:40.161483    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:14:06.231210 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.231172    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:14:06.231752 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.231507    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" podUID="a55b5f98-0962-4090-b42a-b3cb07980875" containerName="payload-processing" containerID="cri-o://0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a" gracePeriod=30
Sep 08 17:14:06.237813 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.237785    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:14:06.238078 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.238034    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" podUID="9a52e54a-3964-4d6e-a991-9f301bd10f58" containerName="payload-pre-processing" containerID="cri-o://f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b" gracePeriod=30
Sep 08 17:14:06.488247 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.488176    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:14:06.524009 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.523984    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:14:06.543431 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.543400    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume\") pod \"a55b5f98-0962-4090-b42a-b3cb07980875\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") "
Sep 08 17:14:06.543564 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.543519    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxgvt\" (UniqueName: \"kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt\") pod \"9a52e54a-3964-4d6e-a991-9f301bd10f58\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") "
Sep 08 17:14:06.543564 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.543551    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume\") pod \"9a52e54a-3964-4d6e-a991-9f301bd10f58\" (UID: \"9a52e54a-3964-4d6e-a991-9f301bd10f58\") "
Sep 08 17:14:06.543682 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.543621    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8stk\" (UniqueName: \"kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk\") pod \"a55b5f98-0962-4090-b42a-b3cb07980875\" (UID: \"a55b5f98-0962-4090-b42a-b3cb07980875\") "
Sep 08 17:14:06.543966 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.543803    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a55b5f98-0962-4090-b42a-b3cb07980875" (UID: "a55b5f98-0962-4090-b42a-b3cb07980875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:14:06.544078 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.544008    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a52e54a-3964-4d6e-a991-9f301bd10f58" (UID: "9a52e54a-3964-4d6e-a991-9f301bd10f58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:14:06.546082 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.546050    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt" (OuterVolumeSpecName: "kube-api-access-fxgvt") pod "9a52e54a-3964-4d6e-a991-9f301bd10f58" (UID: "9a52e54a-3964-4d6e-a991-9f301bd10f58"). InnerVolumeSpecName "kube-api-access-fxgvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:14:06.546200 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.546130    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk" (OuterVolumeSpecName: "kube-api-access-r8stk") pod "a55b5f98-0962-4090-b42a-b3cb07980875" (UID: "a55b5f98-0962-4090-b42a-b3cb07980875"). InnerVolumeSpecName "kube-api-access-r8stk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:14:06.644979 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.644936    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxgvt\" (UniqueName: \"kubernetes.io/projected/9a52e54a-3964-4d6e-a991-9f301bd10f58-kube-api-access-fxgvt\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:06.644979 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.644969    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a52e54a-3964-4d6e-a991-9f301bd10f58-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:06.644979 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.644982    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8stk\" (UniqueName: \"kubernetes.io/projected/a55b5f98-0962-4090-b42a-b3cb07980875-kube-api-access-r8stk\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:06.644979 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:06.644993    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55b5f98-0962-4090-b42a-b3cb07980875-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:07.277662 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.277621    2542 generic.go:356] "Generic (PLEG): container finished" podID="a55b5f98-0962-4090-b42a-b3cb07980875" containerID="0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a" exitCode=0
Sep 08 17:14:07.278106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.277670    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"
Sep 08 17:14:07.278106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.277702    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" event={"ID":"a55b5f98-0962-4090-b42a-b3cb07980875","Type":"ContainerDied","Data":"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"}
Sep 08 17:14:07.278106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.277757    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj" event={"ID":"a55b5f98-0962-4090-b42a-b3cb07980875","Type":"ContainerDied","Data":"0f36733cca30ea70b5f41a7e999d4f98e1432f65da84c428812c1a114d91f2a6"}
Sep 08 17:14:07.278106 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.277788    2542 scope.go:117] "RemoveContainer" containerID="0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"
Sep 08 17:14:07.279176 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.279015    2542 generic.go:356] "Generic (PLEG): container finished" podID="9a52e54a-3964-4d6e-a991-9f301bd10f58" containerID="f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b" exitCode=0
Sep 08 17:14:07.279176 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.279068    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"
Sep 08 17:14:07.279176 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.279102    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" event={"ID":"9a52e54a-3964-4d6e-a991-9f301bd10f58","Type":"ContainerDied","Data":"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"}
Sep 08 17:14:07.279176 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.279144    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk" event={"ID":"9a52e54a-3964-4d6e-a991-9f301bd10f58","Type":"ContainerDied","Data":"aa9e800bafdf7b45e660c7e4fc5c72ff139f1b9fba8cb09ceda32545be813ef3"}
Sep 08 17:14:07.286892 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.286861    2542 scope.go:117] "RemoveContainer" containerID="0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"
Sep 08 17:14:07.287192 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:14:07.287156    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a\": container with ID starting with 0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a not found: ID does not exist" containerID="0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"
Sep 08 17:14:07.287249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.287184    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a"} err="failed to get container status \"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a\": rpc error: code = NotFound desc = could not find container \"0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a\": container with ID starting with 0d749417a854eddf9680d4a11ae3f5668dc3a06a40c0695bbd9d6a29ebc9ee6a not found: ID does not exist"
Sep 08 17:14:07.287249 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.287206    2542 scope.go:117] "RemoveContainer" containerID="f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"
Sep 08 17:14:07.295238 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.295218    2542 scope.go:117] "RemoveContainer" containerID="f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"
Sep 08 17:14:07.295550 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:14:07.295529    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b\": container with ID starting with f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b not found: ID does not exist" containerID="f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"
Sep 08 17:14:07.295629 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.295556    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b"} err="failed to get container status \"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b\": rpc error: code = NotFound desc = could not find container \"f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b\": container with ID starting with f460fd26f6fa4e265d2c2a577c9768b20f7a4cb039fded4e9678f56aa5cc1f6b not found: ID does not exist"
Sep 08 17:14:07.297077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.297056    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:14:07.299241 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.299218    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f0d25d-cf69f568f-fnpnj"]
Sep 08 17:14:07.309510 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.309486    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:14:07.315253 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:07.315231    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f0d25d-64f7cc8bf8-hwfzk"]
Sep 08 17:14:09.116874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:09.116839    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a52e54a-3964-4d6e-a991-9f301bd10f58" path="/var/lib/kubelet/pods/9a52e54a-3964-4d6e-a991-9f301bd10f58/volumes"
Sep 08 17:14:09.117260 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:09.117208    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55b5f98-0962-4090-b42a-b3cb07980875" path="/var/lib/kubelet/pods/a55b5f98-0962-4090-b42a-b3cb07980875/volumes"
Sep 08 17:14:13.310113 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310076    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"]
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310466    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55b5f98-0962-4090-b42a-b3cb07980875" containerName="payload-processing"
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310480    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55b5f98-0962-4090-b42a-b3cb07980875" containerName="payload-processing"
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310494    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a52e54a-3964-4d6e-a991-9f301bd10f58" containerName="payload-pre-processing"
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310501    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a52e54a-3964-4d6e-a991-9f301bd10f58" containerName="payload-pre-processing"
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310571    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a52e54a-3964-4d6e-a991-9f301bd10f58" containerName="payload-pre-processing"
Sep 08 17:14:13.310647 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.310579    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55b5f98-0962-4090-b42a-b3cb07980875" containerName="payload-processing"
Sep 08 17:14:13.315245 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.315224    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:13.317950 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.317925    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:14:13.318137 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.318077    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:14:13.321868 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.321841    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"]
Sep 08 17:14:13.411019 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.410971    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp\" (UID: \"ddaf3e32-11ae-4491-a042-5c3079eb08ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:13.512281 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.512223    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp\" (UID: \"ddaf3e32-11ae-4491-a042-5c3079eb08ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:13.512887 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.512865    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp\" (UID: \"ddaf3e32-11ae-4491-a042-5c3079eb08ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:13.627362 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.627259    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:13.900596 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:13.900491    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"]
Sep 08 17:14:13.902767 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:14:13.902744    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaf3e32_11ae_4491_a042_5c3079eb08ca.slice/crio-fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13 WatchSource:0}: Error finding container fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13: Status 404 returned error can't find the container with id fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13
Sep 08 17:14:14.315410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:14.315349    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp" event={"ID":"ddaf3e32-11ae-4491-a042-5c3079eb08ca","Type":"ContainerStarted","Data":"427fc78fc0c14207e45268d737c8e9fe43e085fb844cf4b1664896602c7776fa"}
Sep 08 17:14:14.315878 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:14.315426    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp" event={"ID":"ddaf3e32-11ae-4491-a042-5c3079eb08ca","Type":"ContainerStarted","Data":"fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13"}
Sep 08 17:14:14.332604 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:14.332552    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp" podStartSLOduration=1.332535166 podStartE2EDuration="1.332535166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:14.330903889 +0000 UTC m=+1397.798110083" watchObservedRunningTime="2026-09-08 17:14:14.332535166 +0000 UTC m=+1397.799741359"
Sep 08 17:14:15.322077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:15.322045    2542 generic.go:356] "Generic (PLEG): container finished" podID="ddaf3e32-11ae-4491-a042-5c3079eb08ca" containerID="427fc78fc0c14207e45268d737c8e9fe43e085fb844cf4b1664896602c7776fa" exitCode=0
Sep 08 17:14:15.322491 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:15.322097    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp" event={"ID":"ddaf3e32-11ae-4491-a042-5c3079eb08ca","Type":"ContainerDied","Data":"427fc78fc0c14207e45268d737c8e9fe43e085fb844cf4b1664896602c7776fa"}
Sep 08 17:14:16.462933 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:16.462909    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:16.536668 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:16.536629    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca\") pod \"ddaf3e32-11ae-4491-a042-5c3079eb08ca\" (UID: \"ddaf3e32-11ae-4491-a042-5c3079eb08ca\") "
Sep 08 17:14:16.536988 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:16.536965    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddaf3e32-11ae-4491-a042-5c3079eb08ca" (UID: "ddaf3e32-11ae-4491-a042-5c3079eb08ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:14:16.637406 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:16.637316    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddaf3e32-11ae-4491-a042-5c3079eb08ca-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:17.333689 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:17.333658    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"
Sep 08 17:14:17.333689 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:17.333671    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp" event={"ID":"ddaf3e32-11ae-4491-a042-5c3079eb08ca","Type":"ContainerDied","Data":"fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13"}
Sep 08 17:14:17.333908 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:17.333715    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbee9781cb65546263ebcbe159459246d377a63c8c5b0e20dc7a3d7521d8f13"
Sep 08 17:14:20.498870 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:20.498829    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:14:21.582398 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:21.582361    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:14:27.104028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.103989    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:14:27.104566 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.104546    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaf3e32-11ae-4491-a042-5c3079eb08ca" containerName="revoke-keys"
Sep 08 17:14:27.104641 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.104568    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaf3e32-11ae-4491-a042-5c3079eb08ca" containerName="revoke-keys"
Sep 08 17:14:27.104709 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.104684    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaf3e32-11ae-4491-a042-5c3079eb08ca" containerName="revoke-keys"
Sep 08 17:14:27.107680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.107659    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.110241 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.110211    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c05c9f\"/\"rate-test-model-c05c9f-kserve-self-signed-certs\""
Sep 08 17:14:27.119404 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.119377    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:14:27.234463 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234421    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.234668 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234548    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.234755 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234732    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.234821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234805    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7znks\" (UniqueName: \"kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.234881 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234868    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.234920 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.234892    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336440 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336398    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336440 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336439    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336737 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336483    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336737 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336512    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336737 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336544    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336737 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336585    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7znks\" (UniqueName: \"kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336959 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336832    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.336959 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336877    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.337030 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.336969    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.338862 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.338832    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.339099 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.339082    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.344947 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.344920    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7znks\" (UniqueName: \"kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks\") pod \"rate-test-model-c05c9f-kserve-6454cbf795-jdq5l\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") " pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.417916 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.417827    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:27.692437 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:27.692410    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:14:27.694278 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:14:27.694251    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4f7d53b_d47d_4a6a_9399_2b880a222efb.slice/crio-392879556af0fe4dcf03c290f54fd3cda05353823be68562dd8ae3696c61d05b WatchSource:0}: Error finding container 392879556af0fe4dcf03c290f54fd3cda05353823be68562dd8ae3696c61d05b: Status 404 returned error can't find the container with id 392879556af0fe4dcf03c290f54fd3cda05353823be68562dd8ae3696c61d05b
Sep 08 17:14:28.387097 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:28.387055    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" event={"ID":"c4f7d53b-d47d-4a6a-9399-2b880a222efb","Type":"ContainerStarted","Data":"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"}
Sep 08 17:14:28.387097 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:28.387099    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" event={"ID":"c4f7d53b-d47d-4a6a-9399-2b880a222efb","Type":"ContainerStarted","Data":"392879556af0fe4dcf03c290f54fd3cda05353823be68562dd8ae3696c61d05b"}
Sep 08 17:14:28.404745 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:28.404691    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" podStartSLOduration=1.404675317 podStartE2EDuration="1.404675317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:28.40380215 +0000 UTC m=+1411.871008342" watchObservedRunningTime="2026-09-08 17:14:28.404675317 +0000 UTC m=+1411.871881509"
Sep 08 17:14:29.162377 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.162335    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:14:29.162628 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.162597    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" podUID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" containerName="payload-processing" containerID="cri-o://a92e0805076c12b872048271e6cfe18259ce06f8961250e994f91593cfd8cfbe" gracePeriod=30
Sep 08 17:14:29.341347 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.341297    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"]
Sep 08 17:14:29.343957 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.343930    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b9f2b8-7k5gp"]
Sep 08 17:14:29.394269 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.394235    2542 generic.go:356] "Generic (PLEG): container finished" podID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" containerID="a92e0805076c12b872048271e6cfe18259ce06f8961250e994f91593cfd8cfbe" exitCode=0
Sep 08 17:14:29.394269 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.394262    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" event={"ID":"74ad4a2f-b719-4a21-ab77-1663a947f5cf","Type":"ContainerDied","Data":"a92e0805076c12b872048271e6cfe18259ce06f8961250e994f91593cfd8cfbe"}
Sep 08 17:14:29.418228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.418165    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:14:29.561900 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.561862    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq8cm\" (UniqueName: \"kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm\") pod \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") "
Sep 08 17:14:29.562074 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.561913    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume\") pod \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\" (UID: \"74ad4a2f-b719-4a21-ab77-1663a947f5cf\") "
Sep 08 17:14:29.562386 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.562359    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74ad4a2f-b719-4a21-ab77-1663a947f5cf" (UID: "74ad4a2f-b719-4a21-ab77-1663a947f5cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:14:29.564076 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.564048    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm" (OuterVolumeSpecName: "kube-api-access-vq8cm") pod "74ad4a2f-b719-4a21-ab77-1663a947f5cf" (UID: "74ad4a2f-b719-4a21-ab77-1663a947f5cf"). InnerVolumeSpecName "kube-api-access-vq8cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:14:29.662772 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.662735    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8cm\" (UniqueName: \"kubernetes.io/projected/74ad4a2f-b719-4a21-ab77-1663a947f5cf-kube-api-access-vq8cm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:29.662772 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:29.662764    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ad4a2f-b719-4a21-ab77-1663a947f5cf-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:30.400314 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:30.400285    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"
Sep 08 17:14:30.400715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:30.400346    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp" event={"ID":"74ad4a2f-b719-4a21-ab77-1663a947f5cf","Type":"ContainerDied","Data":"eb069e63f19e848c36d0a3e12b645d422b80eb61907cd5022838c98588972407"}
Sep 08 17:14:30.400715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:30.400403    2542 scope.go:117] "RemoveContainer" containerID="a92e0805076c12b872048271e6cfe18259ce06f8961250e994f91593cfd8cfbe"
Sep 08 17:14:30.421022 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:30.420995    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:14:30.424796 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:30.424770    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b9f2b8-7f4fbb758c-qw9wp"]
Sep 08 17:14:31.115992 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:31.115961    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" path="/var/lib/kubelet/pods/74ad4a2f-b719-4a21-ab77-1663a947f5cf/volumes"
Sep 08 17:14:31.116294 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:31.116282    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaf3e32-11ae-4491-a042-5c3079eb08ca" path="/var/lib/kubelet/pods/ddaf3e32-11ae-4491-a042-5c3079eb08ca/volumes"
Sep 08 17:14:35.518515 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.518457    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" podUID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" containerName="main" containerID="cri-o://33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074" gracePeriod=45
Sep 08 17:14:35.775270 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.775209    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:14:35.919576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919533    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.919576 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919580    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzftf\" (UniqueName: \"kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.919823 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919635    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.919823 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919651    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.919823 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919699    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.919823 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919745    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache\") pod \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\" (UID: \"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7\") "
Sep 08 17:14:35.920027 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919941    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:35.920027 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.919958    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home" (OuterVolumeSpecName: "home") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:35.920138 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.920101    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:35.920138 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.920108    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:35.920138 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.920121    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:35.921851 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.921825    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm" (OuterVolumeSpecName: "dshm") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:35.921983 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.921869    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:14:35.921983 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:35.921902    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf" (OuterVolumeSpecName: "kube-api-access-vzftf") pod "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" (UID: "d7b1846f-7d58-4a36-9eb2-caca9d29cdd7"). InnerVolumeSpecName "kube-api-access-vzftf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:14:36.021601 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.021541    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:36.021601 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.021593    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:36.021601 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.021609    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:36.021863 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.021633    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzftf\" (UniqueName: \"kubernetes.io/projected/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7-kube-api-access-vzftf\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:36.433260 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.433227    2542 generic.go:356] "Generic (PLEG): container finished" podID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" containerID="33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074" exitCode=0
Sep 08 17:14:36.433459 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.433269    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"
Sep 08 17:14:36.433459 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.433282    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" event={"ID":"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7","Type":"ContainerDied","Data":"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"}
Sep 08 17:14:36.433459 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.433312    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x" event={"ID":"d7b1846f-7d58-4a36-9eb2-caca9d29cdd7","Type":"ContainerDied","Data":"67d1641b90dbdfbb8e407ed81bae722f11d4ca3fac450a58a70882e63fab0380"}
Sep 08 17:14:36.433459 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.433352    2542 scope.go:117] "RemoveContainer" containerID="33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"
Sep 08 17:14:36.442734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.442717    2542 scope.go:117] "RemoveContainer" containerID="33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"
Sep 08 17:14:36.443009 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:14:36.442991    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074\": container with ID starting with 33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074 not found: ID does not exist" containerID="33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"
Sep 08 17:14:36.443059 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.443019    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074"} err="failed to get container status \"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074\": rpc error: code = NotFound desc = could not find container \"33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074\": container with ID starting with 33030fb7158a177af4e3b1c13dd38f1ebcfe4b77bc16c338e8e6a48e2f490074 not found: ID does not exist"
Sep 08 17:14:36.456306 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.456281    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:14:36.462680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.462655    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/sub-test-model-c05c9f-kserve-7f9c554b75-x2z5x"]
Sep 08 17:14:36.482419 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482393    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"]
Sep 08 17:14:36.482740 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482728    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" containerName="main"
Sep 08 17:14:36.482782 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482741    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" containerName="main"
Sep 08 17:14:36.482782 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482759    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" containerName="payload-processing"
Sep 08 17:14:36.482782 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482765    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" containerName="payload-processing"
Sep 08 17:14:36.482874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482842    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" containerName="main"
Sep 08 17:14:36.482874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.482852    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ad4a2f-b719-4a21-ab77-1663a947f5cf" containerName="payload-processing"
Sep 08 17:14:36.487139 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.487122    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:36.489715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.489700    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:14:36.489794 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.489724    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:14:36.493409 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.493387    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"]
Sep 08 17:14:36.603597 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.603549    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" podUID="680c5eae-8162-4d51-8141-6bc590e6443b" containerName="main" containerID="cri-o://5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9" gracePeriod=45
Sep 08 17:14:36.627020 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.626990    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt\" (UID: \"1f5d00d1-af85-435b-91f2-4094a9219f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:36.727734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.727703    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt\" (UID: \"1f5d00d1-af85-435b-91f2-4094a9219f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:36.728316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.728296    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt\" (UID: \"1f5d00d1-af85-435b-91f2-4094a9219f5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:36.797801 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.797767    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:36.869974 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:36.869949    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:14:37.030676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030565    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.030676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030615    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.030676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030644    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tr5q\" (UniqueName: \"kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.030676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030667    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.031008 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030699    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.031008 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030781    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home\") pod \"680c5eae-8162-4d51-8141-6bc590e6443b\" (UID: \"680c5eae-8162-4d51-8141-6bc590e6443b\") "
Sep 08 17:14:37.031008 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.030970    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache" (OuterVolumeSpecName: "model-cache") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:37.031148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.031021    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:37.031148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.031100    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.031148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.031124    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home" (OuterVolumeSpecName: "home") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:37.032825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.032804    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm" (OuterVolumeSpecName: "dshm") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:14:37.032825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.032819    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q" (OuterVolumeSpecName: "kube-api-access-6tr5q") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "kube-api-access-6tr5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:14:37.032940 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.032839    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "680c5eae-8162-4d51-8141-6bc590e6443b" (UID: "680c5eae-8162-4d51-8141-6bc590e6443b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:14:37.075110 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.075084    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"]
Sep 08 17:14:37.077258 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:14:37.077226    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5d00d1_af85_435b_91f2_4094a9219f5e.slice/crio-5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d WatchSource:0}: Error finding container 5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d: Status 404 returned error can't find the container with id 5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d
Sep 08 17:14:37.120568 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.120533    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b1846f-7d58-4a36-9eb2-caca9d29cdd7" path="/var/lib/kubelet/pods/d7b1846f-7d58-4a36-9eb2-caca9d29cdd7/volumes"
Sep 08 17:14:37.131918 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.131891    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680c5eae-8162-4d51-8141-6bc590e6443b-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.131918 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.131918    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.132069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.131927    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tr5q\" (UniqueName: \"kubernetes.io/projected/680c5eae-8162-4d51-8141-6bc590e6443b-kube-api-access-6tr5q\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.132069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.131937    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.132069 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.131946    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680c5eae-8162-4d51-8141-6bc590e6443b-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:37.418154 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.418124    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:37.418302 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.418200    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:37.431483 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.431461    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:37.438733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.438706    2542 generic.go:356] "Generic (PLEG): container finished" podID="680c5eae-8162-4d51-8141-6bc590e6443b" containerID="5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9" exitCode=0
Sep 08 17:14:37.438874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.438756    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"
Sep 08 17:14:37.438874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.438780    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" event={"ID":"680c5eae-8162-4d51-8141-6bc590e6443b","Type":"ContainerDied","Data":"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"}
Sep 08 17:14:37.438874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.438814    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt" event={"ID":"680c5eae-8162-4d51-8141-6bc590e6443b","Type":"ContainerDied","Data":"6ff74b32f483533ef25bc771fcc088c7334825ecab4b69d2c8b92db782ed5d5a"}
Sep 08 17:14:37.438874 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.438833    2542 scope.go:117] "RemoveContainer" containerID="5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"
Sep 08 17:14:37.440658 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.440630    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt" event={"ID":"1f5d00d1-af85-435b-91f2-4094a9219f5e","Type":"ContainerStarted","Data":"1e58b7d7a7800450fa0c19881259cc0e06cc41d7aa29a068aca2b265deea4e0e"}
Sep 08 17:14:37.440813 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.440683    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt" event={"ID":"1f5d00d1-af85-435b-91f2-4094a9219f5e","Type":"ContainerStarted","Data":"5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d"}
Sep 08 17:14:37.447542 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.447517    2542 scope.go:117] "RemoveContainer" containerID="5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"
Sep 08 17:14:37.447856 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:14:37.447822    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9\": container with ID starting with 5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9 not found: ID does not exist" containerID="5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"
Sep 08 17:14:37.447940 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.447865    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9"} err="failed to get container status \"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9\": rpc error: code = NotFound desc = could not find container \"5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9\": container with ID starting with 5b0ae6e97e56d98f23bc4b0f152b73a2335b916820a19830a880bb93cb873fe9 not found: ID does not exist"
Sep 08 17:14:37.454367 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.454348    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:14:37.463750 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.463698    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt" podStartSLOduration=1.463684797 podStartE2EDuration="1.463684797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:37.463163586 +0000 UTC m=+1420.930369779" watchObservedRunningTime="2026-09-08 17:14:37.463684797 +0000 UTC m=+1420.930890989"
Sep 08 17:14:37.476734 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.476710    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:14:37.484070 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:37.484049    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/sub-test-model-338600-kserve-675c7b9659-j5dwt"]
Sep 08 17:14:38.448763 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:38.448727    2542 generic.go:356] "Generic (PLEG): container finished" podID="1f5d00d1-af85-435b-91f2-4094a9219f5e" containerID="1e58b7d7a7800450fa0c19881259cc0e06cc41d7aa29a068aca2b265deea4e0e" exitCode=0
Sep 08 17:14:38.449146 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:38.448813    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt" event={"ID":"1f5d00d1-af85-435b-91f2-4094a9219f5e","Type":"ContainerDied","Data":"1e58b7d7a7800450fa0c19881259cc0e06cc41d7aa29a068aca2b265deea4e0e"}
Sep 08 17:14:39.116825 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:39.116782    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="680c5eae-8162-4d51-8141-6bc590e6443b" path="/var/lib/kubelet/pods/680c5eae-8162-4d51-8141-6bc590e6443b/volumes"
Sep 08 17:14:39.589761 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:39.589734    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:39.754267 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:39.754171    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca\") pod \"1f5d00d1-af85-435b-91f2-4094a9219f5e\" (UID: \"1f5d00d1-af85-435b-91f2-4094a9219f5e\") "
Sep 08 17:14:39.754602 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:39.754576    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f5d00d1-af85-435b-91f2-4094a9219f5e" (UID: "1f5d00d1-af85-435b-91f2-4094a9219f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:14:39.854959 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:39.854921    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f5d00d1-af85-435b-91f2-4094a9219f5e-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:14:40.459613 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:40.459581    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"
Sep 08 17:14:40.459764 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:40.459579    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt" event={"ID":"1f5d00d1-af85-435b-91f2-4094a9219f5e","Type":"ContainerDied","Data":"5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d"}
Sep 08 17:14:40.459764 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:40.459704    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4b4de1df188555e2916fc61aba80a3cb7e4e0679725463aa9c2c032005160d"
Sep 08 17:14:47.208016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:47.207977    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"]
Sep 08 17:14:47.214282 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:47.214251    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ec96ff-phzxt"]
Sep 08 17:14:49.116560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:49.116477    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f5d00d1-af85-435b-91f2-4094a9219f5e" path="/var/lib/kubelet/pods/1f5d00d1-af85-435b-91f2-4094a9219f5e/volumes"
Sep 08 17:14:56.100853 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.100818    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:14:56.101351 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101317    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680c5eae-8162-4d51-8141-6bc590e6443b" containerName="main"
Sep 08 17:14:56.101436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101354    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="680c5eae-8162-4d51-8141-6bc590e6443b" containerName="main"
Sep 08 17:14:56.101436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101398    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5d00d1-af85-435b-91f2-4094a9219f5e" containerName="revoke-keys"
Sep 08 17:14:56.101436 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101408    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5d00d1-af85-435b-91f2-4094a9219f5e" containerName="revoke-keys"
Sep 08 17:14:56.101593 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101511    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="680c5eae-8162-4d51-8141-6bc590e6443b" containerName="main"
Sep 08 17:14:56.101593 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.101524    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5d00d1-af85-435b-91f2-4094a9219f5e" containerName="revoke-keys"
Sep 08 17:14:56.104777 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.104753    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.108554 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.108533    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"openshift-service-ca.crt\""
Sep 08 17:14:56.108680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.108533    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"kube-root-ca.crt\""
Sep 08 17:14:56.108680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.108535    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"rate-test-model-338600-kserve-self-signed-certs\""
Sep 08 17:14:56.108680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.108536    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-338600\"/\"default-dockercfg-4wmmt\""
Sep 08 17:14:56.113222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.112567    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:14:56.191334 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191300    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtdhc\" (UniqueName: \"kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.191502 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191425    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.191558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191512    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.191605 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191587    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.191681 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191653    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.191815 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.191692    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292304 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292262    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtdhc\" (UniqueName: \"kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292316    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292408    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292446    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292484    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292528 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292515    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292869 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292843    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292945 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292849    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.292945 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.292899    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.294667 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.294646    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.294927 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.294905    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.300497 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.300478    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtdhc\" (UniqueName: \"kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc\") pod \"rate-test-model-338600-kserve-655cd7cbb7-fbcvc\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") " pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.416297 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.416204    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:14:56.689111 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:56.689076    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:14:56.692210 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:14:56.692185    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86f9252f_af20_47e7_84dd_9ca0736e6405.slice/crio-eeefb3e11b85474ce9319850a2ca16982b0fe605831c9dfe9ea1b80993f4c861 WatchSource:0}: Error finding container eeefb3e11b85474ce9319850a2ca16982b0fe605831c9dfe9ea1b80993f4c861: Status 404 returned error can't find the container with id eeefb3e11b85474ce9319850a2ca16982b0fe605831c9dfe9ea1b80993f4c861
Sep 08 17:14:57.147821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:57.147737    2542 scope.go:117] "RemoveContainer" containerID="e181a95dcc9e2b4f8f563ed590c5346c0224ccfaccb1aa8d0d1676ccae28970e"
Sep 08 17:14:57.546448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:57.546413    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" event={"ID":"86f9252f-af20-47e7-84dd-9ca0736e6405","Type":"ContainerStarted","Data":"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"}
Sep 08 17:14:57.546448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:57.546452    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" event={"ID":"86f9252f-af20-47e7-84dd-9ca0736e6405","Type":"ContainerStarted","Data":"eeefb3e11b85474ce9319850a2ca16982b0fe605831c9dfe9ea1b80993f4c861"}
Sep 08 17:14:57.562148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:14:57.562095    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" podStartSLOduration=1.562078493 podStartE2EDuration="1.562078493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:14:57.561753124 +0000 UTC m=+1441.028959317" watchObservedRunningTime="2026-09-08 17:14:57.562078493 +0000 UTC m=+1441.029284692"
Sep 08 17:15:00.136052 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.136014    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"]
Sep 08 17:15:00.140883 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.140858    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"]
Sep 08 17:15:00.141033 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.141009    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"
Sep 08 17:15:00.143829 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.143805    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:15:00.144205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.144188    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"
Sep 08 17:15:00.150349 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.150312    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"]
Sep 08 17:15:00.153398 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.153379    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"
Sep 08 17:15:00.158193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.158167    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"]
Sep 08 17:15:00.163013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.162996    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"
Sep 08 17:15:00.437266 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.437241    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"]
Sep 08 17:15:00.439698 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:15:00.439662    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04b5bc8_5a6b_4ce0_8389_e57ff03a1507.slice/crio-505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff WatchSource:0}: Error finding container 505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff: Status 404 returned error can't find the container with id 505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff
Sep 08 17:15:00.441453 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.441429    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"]
Sep 08 17:15:00.442885 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:15:00.442863    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9279ee_05a7_4e4e_b63e_9853b8165e38.slice/crio-e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af WatchSource:0}: Error finding container e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af: Status 404 returned error can't find the container with id e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af
Sep 08 17:15:00.562493 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.562456    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt" event={"ID":"0e9279ee-05a7-4e4e-b63e-9853b8165e38","Type":"ContainerStarted","Data":"e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af"}
Sep 08 17:15:00.563584 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:00.563562    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv" event={"ID":"d04b5bc8-5a6b-4ce0-8389-e57ff03a1507","Type":"ContainerStarted","Data":"505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff"}
Sep 08 17:15:01.573172 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:01.573077    2542 generic.go:356] "Generic (PLEG): container finished" podID="d04b5bc8-5a6b-4ce0-8389-e57ff03a1507" containerID="432af2f96d35ab5ceb052a2ffcc92600cdda59c094c5961c772e5fc90276a4a1" exitCode=0
Sep 08 17:15:01.573639 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:01.573161    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv" event={"ID":"d04b5bc8-5a6b-4ce0-8389-e57ff03a1507","Type":"ContainerDied","Data":"432af2f96d35ab5ceb052a2ffcc92600cdda59c094c5961c772e5fc90276a4a1"}
Sep 08 17:15:01.574609 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:01.574586    2542 generic.go:356] "Generic (PLEG): container finished" podID="0e9279ee-05a7-4e4e-b63e-9853b8165e38" containerID="dd84c69f14b998407529d0c120d3bbbbba0a590aaee4f30aa60522d777f135a0" exitCode=0
Sep 08 17:15:01.574697 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:01.574669    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt" event={"ID":"0e9279ee-05a7-4e4e-b63e-9853b8165e38","Type":"ContainerDied","Data":"dd84c69f14b998407529d0c120d3bbbbba0a590aaee4f30aa60522d777f135a0"}
Sep 08 17:15:02.703914 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:02.703882    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"
Sep 08 17:15:02.738416 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:02.738386    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"
Sep 08 17:15:03.584598 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.584560    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv"
Sep 08 17:15:03.584786 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.584598    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814795-9g5xv" event={"ID":"d04b5bc8-5a6b-4ce0-8389-e57ff03a1507","Type":"ContainerDied","Data":"505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff"}
Sep 08 17:15:03.584786 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.584648    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505f815153c2ea02336da0bf52651ff1be8e12221a2bf72c0d6b5b3e0e5c8eff"
Sep 08 17:15:03.585835 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.585812    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt" event={"ID":"0e9279ee-05a7-4e4e-b63e-9853b8165e38","Type":"ContainerDied","Data":"e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af"}
Sep 08 17:15:03.585944 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.585856    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55395114206413d39c50c5ea34aa9aafca19f53e7c7ba194357d8cb287368af"
Sep 08 17:15:03.585944 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:03.585867    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"
Sep 08 17:15:04.844079 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:04.844046    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:15:04.844516 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:04.844257    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" podUID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" containerName="payload-processing" containerID="cri-o://b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e" gracePeriod=30
Sep 08 17:15:04.854180 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:04.854151    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:15:04.854911 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:04.854409    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" podUID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" containerName="payload-pre-processing" containerID="cri-o://7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6" gracePeriod=30
Sep 08 17:15:05.101883 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.101822    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:15:05.139744 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.139714    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:15:05.174490 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.174459    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnjlj\" (UniqueName: \"kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj\") pod \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") "
Sep 08 17:15:05.174490 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.174495    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwv7w\" (UniqueName: \"kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w\") pod \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") "
Sep 08 17:15:05.174773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.174532    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume\") pod \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\" (UID: \"a065e421-40b6-4fcb-b1b0-42423cf5abd2\") "
Sep 08 17:15:05.174773 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.174566    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume\") pod \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\" (UID: \"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb\") "
Sep 08 17:15:05.174992 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.174964    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a065e421-40b6-4fcb-b1b0-42423cf5abd2" (UID: "a065e421-40b6-4fcb-b1b0-42423cf5abd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:15:05.175060 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.175004    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" (UID: "5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:15:05.176732 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.176703    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj" (OuterVolumeSpecName: "kube-api-access-pnjlj") pod "5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" (UID: "5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb"). InnerVolumeSpecName "kube-api-access-pnjlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:15:05.176732 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.176717    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w" (OuterVolumeSpecName: "kube-api-access-fwv7w") pod "a065e421-40b6-4fcb-b1b0-42423cf5abd2" (UID: "a065e421-40b6-4fcb-b1b0-42423cf5abd2"). InnerVolumeSpecName "kube-api-access-fwv7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:15:05.276388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.276318    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjlj\" (UniqueName: \"kubernetes.io/projected/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-kube-api-access-pnjlj\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:15:05.276388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.276382    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwv7w\" (UniqueName: \"kubernetes.io/projected/a065e421-40b6-4fcb-b1b0-42423cf5abd2-kube-api-access-fwv7w\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:15:05.276388 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.276392    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a065e421-40b6-4fcb-b1b0-42423cf5abd2-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:15:05.276632 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.276401    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:15:05.595808 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.595771    2542 generic.go:356] "Generic (PLEG): container finished" podID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" containerID="b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e" exitCode=0
Sep 08 17:15:05.596013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.595814    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"
Sep 08 17:15:05.596013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.595858    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" event={"ID":"a065e421-40b6-4fcb-b1b0-42423cf5abd2","Type":"ContainerDied","Data":"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"}
Sep 08 17:15:05.596013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.595913    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q" event={"ID":"a065e421-40b6-4fcb-b1b0-42423cf5abd2","Type":"ContainerDied","Data":"b8e6261b3efa201d54a06fe41dde8386472097159acfe2b08c239d7d7b57c010"}
Sep 08 17:15:05.596013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.595943    2542 scope.go:117] "RemoveContainer" containerID="b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"
Sep 08 17:15:05.597221 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.597195    2542 generic.go:356] "Generic (PLEG): container finished" podID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" containerID="7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6" exitCode=0
Sep 08 17:15:05.597316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.597229    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"
Sep 08 17:15:05.597316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.597249    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" event={"ID":"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb","Type":"ContainerDied","Data":"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"}
Sep 08 17:15:05.597316 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.597305    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr" event={"ID":"5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb","Type":"ContainerDied","Data":"e497bcf3323231b592d0c93b9852cb433bb8a9ac460295be25fb3474868340de"}
Sep 08 17:15:05.605385 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.605364    2542 scope.go:117] "RemoveContainer" containerID="b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"
Sep 08 17:15:05.605713 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:15:05.605688    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e\": container with ID starting with b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e not found: ID does not exist" containerID="b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"
Sep 08 17:15:05.605824 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.605717    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e"} err="failed to get container status \"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e\": rpc error: code = NotFound desc = could not find container \"b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e\": container with ID starting with b18c080c755d026b003f5a4c351bfefa73ea243f5eb11391679419ac833be28e not found: ID does not exist"
Sep 08 17:15:05.605824 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.605733    2542 scope.go:117] "RemoveContainer" containerID="7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"
Sep 08 17:15:05.613964 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.613946    2542 scope.go:117] "RemoveContainer" containerID="7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"
Sep 08 17:15:05.614233 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:15:05.614212    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6\": container with ID starting with 7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6 not found: ID does not exist" containerID="7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"
Sep 08 17:15:05.614281 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.614240    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6"} err="failed to get container status \"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6\": rpc error: code = NotFound desc = could not find container \"7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6\": container with ID starting with 7b00911a22b8daeafb7b745b1de80b34b39f370e9cecfea727e8d977b27a18d6 not found: ID does not exist"
Sep 08 17:15:05.621088 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.621063    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:15:05.625767 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.625744    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d7359b-778cfdf6d6-v7nrr"]
Sep 08 17:15:05.637448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.637425    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:15:05.643454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:05.643428    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d7359b-7fb7fd8647-j8j5q"]
Sep 08 17:15:06.416714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:06.416652    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:15:06.416714 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:06.416713    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:15:06.429508 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:06.429477    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:15:06.615041 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:06.615009    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:15:07.116383 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:07.116350    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" path="/var/lib/kubelet/pods/5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb/volumes"
Sep 08 17:15:07.116727 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:07.116715    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" path="/var/lib/kubelet/pods/a065e421-40b6-4fcb-b1b0-42423cf5abd2/volumes"
Sep 08 17:15:53.131729 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:53.131694    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:15:54.185571 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:54.185526    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:15:57.049893 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:57.049867    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:15:57.051916 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:15:57.051892    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:16:08.150970 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.150927    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" podUID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" containerName="main" containerID="cri-o://68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6" gracePeriod=45
Sep 08 17:16:08.394618 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.394594    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:16:08.523779 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523744    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.523779 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523781    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.524015 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523800    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.524015 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523821    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.524015 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523887    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7znks\" (UniqueName: \"kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.524015 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.523908    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir\") pod \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\" (UID: \"c4f7d53b-d47d-4a6a-9399-2b880a222efb\") "
Sep 08 17:16:08.524236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.524085    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home" (OuterVolumeSpecName: "home") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:08.524236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.524192    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:08.524356 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.524240    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:08.526028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.526000    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks" (OuterVolumeSpecName: "kube-api-access-7znks") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "kube-api-access-7znks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:16:08.526028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.526016    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:16:08.526180 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.526009    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm" (OuterVolumeSpecName: "dshm") pod "c4f7d53b-d47d-4a6a-9399-2b880a222efb" (UID: "c4f7d53b-d47d-4a6a-9399-2b880a222efb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:08.625393 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625362    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.625393 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625390    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.625393 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625401    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.625612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625410    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7znks\" (UniqueName: \"kubernetes.io/projected/c4f7d53b-d47d-4a6a-9399-2b880a222efb-kube-api-access-7znks\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.625612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625420    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.625612 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.625438    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f7d53b-d47d-4a6a-9399-2b880a222efb-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:08.896077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.895983    2542 generic.go:356] "Generic (PLEG): container finished" podID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" containerID="68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6" exitCode=0
Sep 08 17:16:08.896077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.896026    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"
Sep 08 17:16:08.896077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.896038    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" event={"ID":"c4f7d53b-d47d-4a6a-9399-2b880a222efb","Type":"ContainerDied","Data":"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"}
Sep 08 17:16:08.896077 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.896070    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l" event={"ID":"c4f7d53b-d47d-4a6a-9399-2b880a222efb","Type":"ContainerDied","Data":"392879556af0fe4dcf03c290f54fd3cda05353823be68562dd8ae3696c61d05b"}
Sep 08 17:16:08.896524 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.896087    2542 scope.go:117] "RemoveContainer" containerID="68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"
Sep 08 17:16:08.905109 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.905091    2542 scope.go:117] "RemoveContainer" containerID="68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"
Sep 08 17:16:08.905381 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:16:08.905363    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6\": container with ID starting with 68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6 not found: ID does not exist" containerID="68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"
Sep 08 17:16:08.905454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.905390    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6"} err="failed to get container status \"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6\": rpc error: code = NotFound desc = could not find container \"68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6\": container with ID starting with 68c28a6052177299709cf42fc6d30776745fcb9bce9be26625b88ba256758fe6 not found: ID does not exist"
Sep 08 17:16:08.918840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.918818    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:16:08.923446 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:08.923426    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c05c9f/rate-test-model-c05c9f-kserve-6454cbf795-jdq5l"]
Sep 08 17:16:09.116637 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.116603    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" path="/var/lib/kubelet/pods/c4f7d53b-d47d-4a6a-9399-2b880a222efb/volumes"
Sep 08 17:16:09.204780 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.204748    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" podUID="86f9252f-af20-47e7-84dd-9ca0736e6405" containerName="main" containerID="cri-o://040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a" gracePeriod=45
Sep 08 17:16:09.459373 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.459284    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:16:09.532674 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532639    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.532674 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532677    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.532928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532700    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.532928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532747    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtdhc\" (UniqueName: \"kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.532928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532764    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.532928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532785    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs\") pod \"86f9252f-af20-47e7-84dd-9ca0736e6405\" (UID: \"86f9252f-af20-47e7-84dd-9ca0736e6405\") "
Sep 08 17:16:09.533123 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.532976    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:09.533123 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.533003    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home" (OuterVolumeSpecName: "home") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:09.533123 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.533069    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache" (OuterVolumeSpecName: "model-cache") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:09.534953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.534927    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:16:09.535063 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.534963    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc" (OuterVolumeSpecName: "kube-api-access-gtdhc") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "kube-api-access-gtdhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:16:09.535063 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.535010    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm" (OuterVolumeSpecName: "dshm") pod "86f9252f-af20-47e7-84dd-9ca0736e6405" (UID: "86f9252f-af20-47e7-84dd-9ca0736e6405"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:16:09.633720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633674    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.633720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633716    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtdhc\" (UniqueName: \"kubernetes.io/projected/86f9252f-af20-47e7-84dd-9ca0736e6405-kube-api-access-gtdhc\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.633720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633727    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.633720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633736    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86f9252f-af20-47e7-84dd-9ca0736e6405-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.633720 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633745    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.634029 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.633753    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86f9252f-af20-47e7-84dd-9ca0736e6405-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:16:09.905551 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.905454    2542 generic.go:356] "Generic (PLEG): container finished" podID="86f9252f-af20-47e7-84dd-9ca0736e6405" containerID="040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a" exitCode=0
Sep 08 17:16:09.905551 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.905498    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"
Sep 08 17:16:09.905551 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.905526    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" event={"ID":"86f9252f-af20-47e7-84dd-9ca0736e6405","Type":"ContainerDied","Data":"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"}
Sep 08 17:16:09.905838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.905564    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc" event={"ID":"86f9252f-af20-47e7-84dd-9ca0736e6405","Type":"ContainerDied","Data":"eeefb3e11b85474ce9319850a2ca16982b0fe605831c9dfe9ea1b80993f4c861"}
Sep 08 17:16:09.905838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.905586    2542 scope.go:117] "RemoveContainer" containerID="040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"
Sep 08 17:16:09.914570 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.914553    2542 scope.go:117] "RemoveContainer" containerID="040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"
Sep 08 17:16:09.914826 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:16:09.914808    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a\": container with ID starting with 040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a not found: ID does not exist" containerID="040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"
Sep 08 17:16:09.914888 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.914832    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a"} err="failed to get container status \"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a\": rpc error: code = NotFound desc = could not find container \"040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a\": container with ID starting with 040df2727ceab2b31ada6e2da9e62ed2652c9b191c693486c404f27a6069b44a not found: ID does not exist"
Sep 08 17:16:09.928444 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.928416    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:16:09.932648 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:09.932623    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-338600/rate-test-model-338600-kserve-655cd7cbb7-fbcvc"]
Sep 08 17:16:11.116582 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:11.116549    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f9252f-af20-47e7-84dd-9ca0736e6405" path="/var/lib/kubelet/pods/86f9252f-af20-47e7-84dd-9ca0736e6405/volumes"
Sep 08 17:16:29.285502 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285464    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285802    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" containerName="payload-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285814    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" containerName="payload-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285823    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f9252f-af20-47e7-84dd-9ca0736e6405" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285828    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f9252f-af20-47e7-84dd-9ca0736e6405" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285835    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04b5bc8-5a6b-4ce0-8389-e57ff03a1507" containerName="cleanup"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285841    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04b5bc8-5a6b-4ce0-8389-e57ff03a1507" containerName="cleanup"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285862    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285867    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285872    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9279ee-05a7-4e4e-b63e-9853b8165e38" containerName="cleanup"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285877    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9279ee-05a7-4e4e-b63e-9853b8165e38" containerName="cleanup"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285885    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" containerName="payload-pre-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285890    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" containerName="payload-pre-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285940    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9279ee-05a7-4e4e-b63e-9853b8165e38" containerName="cleanup"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285948    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="a065e421-40b6-4fcb-b1b0-42423cf5abd2" containerName="payload-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285953    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f7d53b-d47d-4a6a-9399-2b880a222efb" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285959    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecb5a97-c70e-4fc9-a2b1-aa7a981dc4fb" containerName="payload-pre-processing"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285966    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f9252f-af20-47e7-84dd-9ca0736e6405" containerName="main"
Sep 08 17:16:29.285969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.285974    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04b5bc8-5a6b-4ce0-8389-e57ff03a1507" containerName="cleanup"
Sep 08 17:16:29.288901 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.288884    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.292663 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.292640    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a78a1e\""
Sep 08 17:16:29.292899 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.292883    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a78a1e-dockercfg-4skjp\""
Sep 08 17:16:29.299177 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.299151    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:16:29.326258 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.326218    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:16:29.329479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.329457    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.340621 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.340600    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:16:29.404149 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.404115    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qq7\" (UniqueName: \"kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.404360 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.404174    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.404360 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.404273    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qm8\" (UniqueName: \"kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.404360 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.404345    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.504963 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.504922    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qm8\" (UniqueName: \"kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.505119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.504972    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.505119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.505000    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qq7\" (UniqueName: \"kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.505119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.505048    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.505807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.505788    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.505863 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.505788    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.513767 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.513740    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qm8\" (UniqueName: \"kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8\") pod \"payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.513876 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.513806    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qq7\" (UniqueName: \"kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7\") pod \"payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.600019 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.599925    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:29.640178 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.640144    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:29.881898 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.881823    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:16:29.885311 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:16:29.885285    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e000fd5_b6cc_46da_957b_eaa144df1860.slice/crio-0a3f675dae744979575e0ff6eb5e4a80ebf899a40600881f5b14e27db7fc0b98 WatchSource:0}: Error finding container 0a3f675dae744979575e0ff6eb5e4a80ebf899a40600881f5b14e27db7fc0b98: Status 404 returned error can't find the container with id 0a3f675dae744979575e0ff6eb5e4a80ebf899a40600881f5b14e27db7fc0b98
Sep 08 17:16:29.918672 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:29.918645    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:16:29.921135 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:16:29.921110    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b2f8f2_18c4_4e39_ab9a_28102e2e6730.slice/crio-52ab39f2c7ae05e80f392dcbdfa18574351366a55472ecc637c4b95e57678f71 WatchSource:0}: Error finding container 52ab39f2c7ae05e80f392dcbdfa18574351366a55472ecc637c4b95e57678f71: Status 404 returned error can't find the container with id 52ab39f2c7ae05e80f392dcbdfa18574351366a55472ecc637c4b95e57678f71
Sep 08 17:16:30.007074 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.007042    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" event={"ID":"0e000fd5-b6cc-46da-957b-eaa144df1860","Type":"ContainerStarted","Data":"e80d9aaf2f6132586e4cb8616db2cf37646fc4327089dafe515d149372dc4f00"}
Sep 08 17:16:30.007248 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.007099    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" event={"ID":"0e000fd5-b6cc-46da-957b-eaa144df1860","Type":"ContainerStarted","Data":"0a3f675dae744979575e0ff6eb5e4a80ebf899a40600881f5b14e27db7fc0b98"}
Sep 08 17:16:30.007248 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.007173    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:30.008709 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.008683    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" event={"ID":"56b2f8f2-18c4-4e39-ab9a-28102e2e6730","Type":"ContainerStarted","Data":"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"}
Sep 08 17:16:30.008832 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.008729    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:30.008832 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.008751    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" event={"ID":"56b2f8f2-18c4-4e39-ab9a-28102e2e6730","Type":"ContainerStarted","Data":"52ab39f2c7ae05e80f392dcbdfa18574351366a55472ecc637c4b95e57678f71"}
Sep 08 17:16:30.023846 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.023553    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" podStartSLOduration=1.023539008 podStartE2EDuration="1.023539008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:30.02296818 +0000 UTC m=+1533.490174374" watchObservedRunningTime="2026-09-08 17:16:30.023539008 +0000 UTC m=+1533.490745208"
Sep 08 17:16:30.038969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:30.038907    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" podStartSLOduration=1.038889818 podStartE2EDuration="1.038889818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:16:30.037063755 +0000 UTC m=+1533.504269949" watchObservedRunningTime="2026-09-08 17:16:30.038889818 +0000 UTC m=+1533.506096062"
Sep 08 17:16:41.016858 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:41.016825    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:16:41.017264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:41.016898    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:16:57.627585 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:16:57.627508    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v_56b2f8f2-18c4-4e39-ab9a-28102e2e6730/payload-processing/0.log"
Sep 08 17:17:10.816497 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.816458    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:17:10.820003 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.819985    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.822883 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.822857    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-978af9\"/\"ipp-route-978af9-kserve-self-signed-certs\""
Sep 08 17:17:10.823968 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.823949    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-978af9\"/\"openshift-service-ca.crt\""
Sep 08 17:17:10.824194 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.824175    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-978af9\"/\"kube-root-ca.crt\""
Sep 08 17:17:10.824275 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.824186    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-978af9\"/\"default-dockercfg-94m8s\""
Sep 08 17:17:10.829421 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.829397    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:17:10.937879 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.937841    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.938065 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.937893    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.938065 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.937956    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.938065 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.938004    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.938065 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.938023    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl572\" (UniqueName: \"kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:10.938065 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:10.938052    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.038928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.038893    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.038928 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.038929    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039215 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.038946    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl572\" (UniqueName: \"kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039215 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.038972    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039215 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.039020    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039215 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.039053    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039469 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.039357    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039469 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.039400    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.039469 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.039444    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.041276 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.041248    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.041550 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.041535    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.047049 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.047024    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl572\" (UniqueName: \"kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572\") pod \"ipp-route-978af9-kserve-7f78d8555-lvhgk\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") " pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.129646 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.129548    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:11.400701 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:11.400679    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:17:11.402751 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:17:11.402721    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4e2c179_fe20_4766_aca5_1ef879040f60.slice/crio-f1d53c6befb27dbeb6685296dbd98cb0c92309e1b45fbe53cfee5ac81af90b3f WatchSource:0}: Error finding container f1d53c6befb27dbeb6685296dbd98cb0c92309e1b45fbe53cfee5ac81af90b3f: Status 404 returned error can't find the container with id f1d53c6befb27dbeb6685296dbd98cb0c92309e1b45fbe53cfee5ac81af90b3f
Sep 08 17:17:12.217854 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:12.217820    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" event={"ID":"d4e2c179-fe20-4766-aca5-1ef879040f60","Type":"ContainerStarted","Data":"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"}
Sep 08 17:17:12.217854 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:12.217861    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" event={"ID":"d4e2c179-fe20-4766-aca5-1ef879040f60","Type":"ContainerStarted","Data":"f1d53c6befb27dbeb6685296dbd98cb0c92309e1b45fbe53cfee5ac81af90b3f"}
Sep 08 17:17:12.243599 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:12.243550    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" podStartSLOduration=2.243534106 podStartE2EDuration="2.243534106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:17:12.242571457 +0000 UTC m=+1575.709777652" watchObservedRunningTime="2026-09-08 17:17:12.243534106 +0000 UTC m=+1575.710740296"
Sep 08 17:17:21.130446 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:21.130413    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:21.130818 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:21.130616    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:21.143076 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:21.143051    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:21.271666 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:21.271632    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:17:35.603911 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:35.603880    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v_56b2f8f2-18c4-4e39-ab9a-28102e2e6730/payload-processing/0.log"
Sep 08 17:17:40.909098 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:40.909064    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:17:40.912490 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:40.912474    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:40.915055 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:40.915038    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-05b1b1-dockercfg-gx5bt\""
Sep 08 17:17:40.915150 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:40.915089    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-05b1b1\""
Sep 08 17:17:40.924757 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:40.924737    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:17:41.007699 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.007665    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.007871 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.007813    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5trb\" (UniqueName: \"kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.108475 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.108435    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5trb\" (UniqueName: \"kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.108475 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.108476    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.109139 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.109118    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.117692 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.117671    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5trb\" (UniqueName: \"kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb\") pod \"payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.223446 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.223412    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:41.499799 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:41.499771    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:17:41.501357 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:17:41.501315    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5944d9_5515_44b0_ae67_fb6ae23f858c.slice/crio-8214613534b4b3fe820895d7473d77cfa9ee555bf4ea2c90358c10b092df4bda WatchSource:0}: Error finding container 8214613534b4b3fe820895d7473d77cfa9ee555bf4ea2c90358c10b092df4bda: Status 404 returned error can't find the container with id 8214613534b4b3fe820895d7473d77cfa9ee555bf4ea2c90358c10b092df4bda
Sep 08 17:17:42.358729 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:42.358690    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" event={"ID":"3e5944d9-5515-44b0-ae67-fb6ae23f858c","Type":"ContainerStarted","Data":"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"}
Sep 08 17:17:42.359151 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:42.358758    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" event={"ID":"3e5944d9-5515-44b0-ae67-fb6ae23f858c","Type":"ContainerStarted","Data":"8214613534b4b3fe820895d7473d77cfa9ee555bf4ea2c90358c10b092df4bda"}
Sep 08 17:17:42.359151 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:42.358795    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:17:42.378900 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:42.378851    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" podStartSLOduration=2.378834959 podStartE2EDuration="2.378834959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:17:42.376531112 +0000 UTC m=+1605.843737306" watchObservedRunningTime="2026-09-08 17:17:42.378834959 +0000 UTC m=+1605.846041545"
Sep 08 17:17:53.365544 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:17:53.365460    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:18:04.612822 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.612784    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"]
Sep 08 17:18:04.616354 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.616319    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:04.618992 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.618970    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:18:04.619067 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.619005    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:18:04.623772 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.623738    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"]
Sep 08 17:18:04.707451 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.707410    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt\" (UID: \"baaa9343-52ee-4024-8c49-f84ca3d4ca08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:04.808800 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.808758    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt\" (UID: \"baaa9343-52ee-4024-8c49-f84ca3d4ca08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:04.809519 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.809501    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt\" (UID: \"baaa9343-52ee-4024-8c49-f84ca3d4ca08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:04.928108 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:04.928072    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:05.198102 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:05.198076    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"]
Sep 08 17:18:05.200353 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:18:05.200310    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaa9343_52ee_4024_8c49_f84ca3d4ca08.slice/crio-fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e WatchSource:0}: Error finding container fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e: Status 404 returned error can't find the container with id fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e
Sep 08 17:18:05.467566 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:05.467484    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt" event={"ID":"baaa9343-52ee-4024-8c49-f84ca3d4ca08","Type":"ContainerStarted","Data":"cdd715068d26c0a562f17e3e8f78626775c8732183085a4a1326158c2fbf09d3"}
Sep 08 17:18:05.467566 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:05.467540    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt" event={"ID":"baaa9343-52ee-4024-8c49-f84ca3d4ca08","Type":"ContainerStarted","Data":"fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e"}
Sep 08 17:18:05.485486 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:05.485437    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt" podStartSLOduration=1.48542363 podStartE2EDuration="1.48542363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:18:05.483618957 +0000 UTC m=+1628.950825153" watchObservedRunningTime="2026-09-08 17:18:05.48542363 +0000 UTC m=+1628.952629823"
Sep 08 17:18:06.477013 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:06.476981    2542 generic.go:356] "Generic (PLEG): container finished" podID="baaa9343-52ee-4024-8c49-f84ca3d4ca08" containerID="cdd715068d26c0a562f17e3e8f78626775c8732183085a4a1326158c2fbf09d3" exitCode=0
Sep 08 17:18:06.477434 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:06.477048    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt" event={"ID":"baaa9343-52ee-4024-8c49-f84ca3d4ca08","Type":"ContainerDied","Data":"cdd715068d26c0a562f17e3e8f78626775c8732183085a4a1326158c2fbf09d3"}
Sep 08 17:18:07.612214 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:07.612183    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:07.733006 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:07.732914    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca\") pod \"baaa9343-52ee-4024-8c49-f84ca3d4ca08\" (UID: \"baaa9343-52ee-4024-8c49-f84ca3d4ca08\") "
Sep 08 17:18:07.733301 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:07.733277    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "baaa9343-52ee-4024-8c49-f84ca3d4ca08" (UID: "baaa9343-52ee-4024-8c49-f84ca3d4ca08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:18:07.834676 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:07.834642    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/baaa9343-52ee-4024-8c49-f84ca3d4ca08-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:08.487401 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:08.487370    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"
Sep 08 17:18:08.487401 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:08.487393    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt" event={"ID":"baaa9343-52ee-4024-8c49-f84ca3d4ca08","Type":"ContainerDied","Data":"fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e"}
Sep 08 17:18:08.487616 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:08.487439    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fefbb264e45ca055987d8914059180e0f6c790ae0c80996fcc07e375341df71e"
Sep 08 17:18:10.987209 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:10.987170    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:18:10.987587 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:10.987450    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" podUID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" containerName="payload-pre-processing" containerID="cri-o://88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf" gracePeriod=30
Sep 08 17:18:11.200450 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.200416    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"]
Sep 08 17:18:11.204204 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.204177    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-05b1b1-5g4zt"]
Sep 08 17:18:11.228920 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.228887    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:18:11.266753 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.266667    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5trb\" (UniqueName: \"kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb\") pod \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") "
Sep 08 17:18:11.266753 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.266711    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume\") pod \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\" (UID: \"3e5944d9-5515-44b0-ae67-fb6ae23f858c\") "
Sep 08 17:18:11.267107 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.267079    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e5944d9-5515-44b0-ae67-fb6ae23f858c" (UID: "3e5944d9-5515-44b0-ae67-fb6ae23f858c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:18:11.268873 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.268846    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb" (OuterVolumeSpecName: "kube-api-access-v5trb") pod "3e5944d9-5515-44b0-ae67-fb6ae23f858c" (UID: "3e5944d9-5515-44b0-ae67-fb6ae23f858c"). InnerVolumeSpecName "kube-api-access-v5trb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:18:11.368133 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.368095    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5trb\" (UniqueName: \"kubernetes.io/projected/3e5944d9-5515-44b0-ae67-fb6ae23f858c-kube-api-access-v5trb\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:11.368133 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.368124    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e5944d9-5515-44b0-ae67-fb6ae23f858c-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:11.504730 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.504695    2542 generic.go:356] "Generic (PLEG): container finished" podID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" containerID="88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf" exitCode=0
Sep 08 17:18:11.504902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.504745    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"
Sep 08 17:18:11.504902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.504787    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" event={"ID":"3e5944d9-5515-44b0-ae67-fb6ae23f858c","Type":"ContainerDied","Data":"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"}
Sep 08 17:18:11.504902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.504845    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb" event={"ID":"3e5944d9-5515-44b0-ae67-fb6ae23f858c","Type":"ContainerDied","Data":"8214613534b4b3fe820895d7473d77cfa9ee555bf4ea2c90358c10b092df4bda"}
Sep 08 17:18:11.504902 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.504872    2542 scope.go:117] "RemoveContainer" containerID="88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"
Sep 08 17:18:11.514254 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.514233    2542 scope.go:117] "RemoveContainer" containerID="88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"
Sep 08 17:18:11.514559 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:18:11.514541    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf\": container with ID starting with 88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf not found: ID does not exist" containerID="88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"
Sep 08 17:18:11.514611 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.514575    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf"} err="failed to get container status \"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf\": rpc error: code = NotFound desc = could not find container \"88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf\": container with ID starting with 88ed4010352743d373ad98e8890c11023cd9772bc1de2142c86cf029c0c34edf not found: ID does not exist"
Sep 08 17:18:11.529592 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.529536    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:18:11.533243 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:11.533219    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-05b1b1-67cc657478-kg9xb"]
Sep 08 17:18:13.116921 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:13.116889    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" path="/var/lib/kubelet/pods/3e5944d9-5515-44b0-ae67-fb6ae23f858c/volumes"
Sep 08 17:18:13.117381 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:13.117195    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baaa9343-52ee-4024-8c49-f84ca3d4ca08" path="/var/lib/kubelet/pods/baaa9343-52ee-4024-8c49-f84ca3d4ca08/volumes"
Sep 08 17:18:21.727870 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.727833    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"]
Sep 08 17:18:21.728236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728168    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baaa9343-52ee-4024-8c49-f84ca3d4ca08" containerName="revoke-keys"
Sep 08 17:18:21.728236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728179    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="baaa9343-52ee-4024-8c49-f84ca3d4ca08" containerName="revoke-keys"
Sep 08 17:18:21.728236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728199    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" containerName="payload-pre-processing"
Sep 08 17:18:21.728236 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728205    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" containerName="payload-pre-processing"
Sep 08 17:18:21.728402 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728259    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e5944d9-5515-44b0-ae67-fb6ae23f858c" containerName="payload-pre-processing"
Sep 08 17:18:21.728402 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.728267    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="baaa9343-52ee-4024-8c49-f84ca3d4ca08" containerName="revoke-keys"
Sep 08 17:18:21.732517 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.732500    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:21.735090 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.735072    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 17:18:21.735195 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.735167    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hfwpm\""
Sep 08 17:18:21.739154 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.739130    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"]
Sep 08 17:18:21.853838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.853807    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv\" (UID: \"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:21.954595 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.954553    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv\" (UID: \"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:21.955182 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:21.955161    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv\" (UID: \"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:22.043954 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:22.043870    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:22.334479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:22.334441    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"]
Sep 08 17:18:22.337141 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:18:22.337113    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2cc50a_22fa_4c03_83bc_1e256e1ce56d.slice/crio-b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44 WatchSource:0}: Error finding container b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44: Status 404 returned error can't find the container with id b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44
Sep 08 17:18:22.561217 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:22.561187    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv" event={"ID":"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d","Type":"ContainerStarted","Data":"2c1ca63c7e0bde858505b98462ca80e7b999c7cce10e2a571e5a216e8aebb3e9"}
Sep 08 17:18:22.561361 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:22.561240    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv" event={"ID":"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d","Type":"ContainerStarted","Data":"b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44"}
Sep 08 17:18:22.581150 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:22.581110    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv" podStartSLOduration=1.581096521 podStartE2EDuration="1.581096521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:18:22.578986732 +0000 UTC m=+1646.046192927" watchObservedRunningTime="2026-09-08 17:18:22.581096521 +0000 UTC m=+1646.048302758"
Sep 08 17:18:23.567450 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:23.567413    2542 generic.go:356] "Generic (PLEG): container finished" podID="4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" containerID="2c1ca63c7e0bde858505b98462ca80e7b999c7cce10e2a571e5a216e8aebb3e9" exitCode=0
Sep 08 17:18:23.567904 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:23.567493    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv" event={"ID":"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d","Type":"ContainerDied","Data":"2c1ca63c7e0bde858505b98462ca80e7b999c7cce10e2a571e5a216e8aebb3e9"}
Sep 08 17:18:24.701062 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:24.701039    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:24.778438 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:24.778399    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca\") pod \"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d\" (UID: \"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d\") "
Sep 08 17:18:24.778771 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:24.778748    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" (UID: "4e2cc50a-22fa-4c03-83bc-1e256e1ce56d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:18:24.879628 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:24.879535    2542 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d-maas-api-service-ca\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:25.578208 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:25.578171    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv" event={"ID":"4e2cc50a-22fa-4c03-83bc-1e256e1ce56d","Type":"ContainerDied","Data":"b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44"}
Sep 08 17:18:25.578208 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:25.578196    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"
Sep 08 17:18:25.578490 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:25.578230    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5d1bfcae09f7aa4bdaecf55440edb97a6b1ebcd8e3bd880ecb77eaed72e2f44"
Sep 08 17:18:37.672256 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:37.672217    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"]
Sep 08 17:18:37.675489 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:37.675466    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-978af9-fbdlv"]
Sep 08 17:18:39.115931 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:39.115876    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" path="/var/lib/kubelet/pods/4e2cc50a-22fa-4c03-83bc-1e256e1ce56d/volumes"
Sep 08 17:18:43.000260 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:43.000211    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:18:58.021122 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.021059    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" podUID="d4e2c179-fe20-4766-aca5-1ef879040f60" containerName="main" containerID="cri-o://ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360" gracePeriod=45
Sep 08 17:18:58.267951 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.267926    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:18:58.371805 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371717    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.371805 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371793    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl572\" (UniqueName: \"kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.372046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371814    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.372046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371837    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.372046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371853    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.372046 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.371885    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir\") pod \"d4e2c179-fe20-4766-aca5-1ef879040f60\" (UID: \"d4e2c179-fe20-4766-aca5-1ef879040f60\") "
Sep 08 17:18:58.372253 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.372137    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home" (OuterVolumeSpecName: "home") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:18:58.372253 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.372151    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:18:58.372253 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.372176    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:18:58.374050 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.374028    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572" (OuterVolumeSpecName: "kube-api-access-pl572") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "kube-api-access-pl572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:18:58.374299 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.374279    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm" (OuterVolumeSpecName: "dshm") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:18:58.374299 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.374279    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4e2c179-fe20-4766-aca5-1ef879040f60" (UID: "d4e2c179-fe20-4766-aca5-1ef879040f60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:18:58.473427 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473393    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl572\" (UniqueName: \"kubernetes.io/projected/d4e2c179-fe20-4766-aca5-1ef879040f60-kube-api-access-pl572\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.473427 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473421    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4e2c179-fe20-4766-aca5-1ef879040f60-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.473427 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473430    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.473427 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473438    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.473707 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473446    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.473707 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.473453    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4e2c179-fe20-4766-aca5-1ef879040f60-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:18:58.739489 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.739452    2542 generic.go:356] "Generic (PLEG): container finished" podID="d4e2c179-fe20-4766-aca5-1ef879040f60" containerID="ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360" exitCode=0
Sep 08 17:18:58.739640 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.739495    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"
Sep 08 17:18:58.739640 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.739496    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" event={"ID":"d4e2c179-fe20-4766-aca5-1ef879040f60","Type":"ContainerDied","Data":"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"}
Sep 08 17:18:58.739640 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.739622    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk" event={"ID":"d4e2c179-fe20-4766-aca5-1ef879040f60","Type":"ContainerDied","Data":"f1d53c6befb27dbeb6685296dbd98cb0c92309e1b45fbe53cfee5ac81af90b3f"}
Sep 08 17:18:58.739815 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.739649    2542 scope.go:117] "RemoveContainer" containerID="ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"
Sep 08 17:18:58.749048 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.749031    2542 scope.go:117] "RemoveContainer" containerID="ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"
Sep 08 17:18:58.749377 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:18:58.749345    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360\": container with ID starting with ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360 not found: ID does not exist" containerID="ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"
Sep 08 17:18:58.749443 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.749385    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360"} err="failed to get container status \"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360\": rpc error: code = NotFound desc = could not find container \"ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360\": container with ID starting with ab4a2d800e3a20eea9c38aa9c77d88a230bcd313e7974639d7bc97b7abc14360 not found: ID does not exist"
Sep 08 17:18:58.764048 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.764021    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:18:58.768011 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:58.767989    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-978af9/ipp-route-978af9-kserve-7f78d8555-lvhgk"]
Sep 08 17:18:59.116952 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:18:59.116868    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e2c179-fe20-4766-aca5-1ef879040f60" path="/var/lib/kubelet/pods/d4e2c179-fe20-4766-aca5-1ef879040f60/volumes"
Sep 08 17:19:37.802148 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.802105    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:19:37.809120 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.809086    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:19:37.809372 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.809340    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" podUID="0e000fd5-b6cc-46da-957b-eaa144df1860" containerName="payload-pre-processing" containerID="cri-o://e80d9aaf2f6132586e4cb8616db2cf37646fc4327089dafe515d149372dc4f00" gracePeriod=30
Sep 08 17:19:37.948232 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.948202    2542 generic.go:356] "Generic (PLEG): container finished" podID="0e000fd5-b6cc-46da-957b-eaa144df1860" containerID="e80d9aaf2f6132586e4cb8616db2cf37646fc4327089dafe515d149372dc4f00" exitCode=0
Sep 08 17:19:37.948423 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.948278    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" event={"ID":"0e000fd5-b6cc-46da-957b-eaa144df1860","Type":"ContainerDied","Data":"e80d9aaf2f6132586e4cb8616db2cf37646fc4327089dafe515d149372dc4f00"}
Sep 08 17:19:37.948481 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:37.948434    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" podUID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" containerName="payload-processing" containerID="cri-o://3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98" gracePeriod=30
Sep 08 17:19:38.069001 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.068973    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:19:38.127568 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.126907    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9qq7\" (UniqueName: \"kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7\") pod \"0e000fd5-b6cc-46da-957b-eaa144df1860\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") "
Sep 08 17:19:38.127568 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.126992    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume\") pod \"0e000fd5-b6cc-46da-957b-eaa144df1860\" (UID: \"0e000fd5-b6cc-46da-957b-eaa144df1860\") "
Sep 08 17:19:38.127805 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.127635    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e000fd5-b6cc-46da-957b-eaa144df1860" (UID: "0e000fd5-b6cc-46da-957b-eaa144df1860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:19:38.129961 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.129929    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7" (OuterVolumeSpecName: "kube-api-access-c9qq7") pod "0e000fd5-b6cc-46da-957b-eaa144df1860" (UID: "0e000fd5-b6cc-46da-957b-eaa144df1860"). InnerVolumeSpecName "kube-api-access-c9qq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:19:38.179465 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.179438    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:19:38.228371 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228314    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume\") pod \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") "
Sep 08 17:19:38.228560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228482    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55qm8\" (UniqueName: \"kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8\") pod \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\" (UID: \"56b2f8f2-18c4-4e39-ab9a-28102e2e6730\") "
Sep 08 17:19:38.228680 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228650    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56b2f8f2-18c4-4e39-ab9a-28102e2e6730" (UID: "56b2f8f2-18c4-4e39-ab9a-28102e2e6730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:19:38.228827 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228798    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:38.228827 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228819    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e000fd5-b6cc-46da-957b-eaa144df1860-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:38.228827 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.228828    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qq7\" (UniqueName: \"kubernetes.io/projected/0e000fd5-b6cc-46da-957b-eaa144df1860-kube-api-access-c9qq7\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:38.230474 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.230451    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8" (OuterVolumeSpecName: "kube-api-access-55qm8") pod "56b2f8f2-18c4-4e39-ab9a-28102e2e6730" (UID: "56b2f8f2-18c4-4e39-ab9a-28102e2e6730"). InnerVolumeSpecName "kube-api-access-55qm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:19:38.329553 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.329463    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55qm8\" (UniqueName: \"kubernetes.io/projected/56b2f8f2-18c4-4e39-ab9a-28102e2e6730-kube-api-access-55qm8\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:38.955754 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.955720    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"
Sep 08 17:19:38.956244 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.955714    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm" event={"ID":"0e000fd5-b6cc-46da-957b-eaa144df1860","Type":"ContainerDied","Data":"0a3f675dae744979575e0ff6eb5e4a80ebf899a40600881f5b14e27db7fc0b98"}
Sep 08 17:19:38.956244 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.955915    2542 scope.go:117] "RemoveContainer" containerID="e80d9aaf2f6132586e4cb8616db2cf37646fc4327089dafe515d149372dc4f00"
Sep 08 17:19:38.957552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.957288    2542 generic.go:356] "Generic (PLEG): container finished" podID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" containerID="3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98" exitCode=0
Sep 08 17:19:38.957552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.957368    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"
Sep 08 17:19:38.957552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.957364    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" event={"ID":"56b2f8f2-18c4-4e39-ab9a-28102e2e6730","Type":"ContainerDied","Data":"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"}
Sep 08 17:19:38.957552 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.957487    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v" event={"ID":"56b2f8f2-18c4-4e39-ab9a-28102e2e6730","Type":"ContainerDied","Data":"52ab39f2c7ae05e80f392dcbdfa18574351366a55472ecc637c4b95e57678f71"}
Sep 08 17:19:38.966661 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.966638    2542 scope.go:117] "RemoveContainer" containerID="3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"
Sep 08 17:19:38.974687 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.974670    2542 scope.go:117] "RemoveContainer" containerID="3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"
Sep 08 17:19:38.974933 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:19:38.974914    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98\": container with ID starting with 3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98 not found: ID does not exist" containerID="3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"
Sep 08 17:19:38.974976 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.974942    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98"} err="failed to get container status \"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98\": rpc error: code = NotFound desc = could not find container \"3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98\": container with ID starting with 3c59eaf14334b25735d1698b30ab2b6de6ff70afa2eeba4c7cdd755398fe6d98 not found: ID does not exist"
Sep 08 17:19:38.979033 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.979006    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:19:38.983711 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.983689    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a78a1e-5745685fdb-hnwbm"]
Sep 08 17:19:38.999484 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:38.999457    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:19:39.005892 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:39.005870    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a78a1e-f5f64cd6f-dwp9v"]
Sep 08 17:19:39.117704 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:39.117622    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e000fd5-b6cc-46da-957b-eaa144df1860" path="/var/lib/kubelet/pods/0e000fd5-b6cc-46da-957b-eaa144df1860/volumes"
Sep 08 17:19:39.117980 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:39.117967    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" path="/var/lib/kubelet/pods/56b2f8f2-18c4-4e39-ab9a-28102e2e6730/volumes"
Sep 08 17:19:55.742675 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.742638    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"]
Sep 08 17:19:55.745226 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.745204    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c05c9f-29814795-sqmtt"]
Sep 08 17:19:55.771192 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.771155    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:19:55.771449 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.771391    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" podUID="903344ac-7a1a-44e1-901d-4eeec854c8cf" containerName="payload-processing" containerID="cri-o://ee1f9981c8c376ef3cc7b8b30827876635b0f9b13db1b6cee64e3ac5cf9dec29" gracePeriod=30
Sep 08 17:19:55.785787 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.785755    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:19:55.785963 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:55.785942    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" podUID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" containerName="payload-pre-processing" containerID="cri-o://0e01010820413ed7578f35c23cf0d2aacd42a72b695a656aa30c9f5be65dbc11" gracePeriod=30
Sep 08 17:19:56.044806 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.044775    2542 generic.go:356] "Generic (PLEG): container finished" podID="903344ac-7a1a-44e1-901d-4eeec854c8cf" containerID="ee1f9981c8c376ef3cc7b8b30827876635b0f9b13db1b6cee64e3ac5cf9dec29" exitCode=0
Sep 08 17:19:56.044946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.044809    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" event={"ID":"903344ac-7a1a-44e1-901d-4eeec854c8cf","Type":"ContainerDied","Data":"ee1f9981c8c376ef3cc7b8b30827876635b0f9b13db1b6cee64e3ac5cf9dec29"}
Sep 08 17:19:56.044946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.044867    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb" event={"ID":"903344ac-7a1a-44e1-901d-4eeec854c8cf","Type":"ContainerDied","Data":"d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5"}
Sep 08 17:19:56.044946 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.044885    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18ca50459b21f11c98f9e3b95e63629075372ba74f18eabdb24e29295db67f5"
Sep 08 17:19:56.046233 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.046211    2542 generic.go:356] "Generic (PLEG): container finished" podID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" containerID="0e01010820413ed7578f35c23cf0d2aacd42a72b695a656aa30c9f5be65dbc11" exitCode=0
Sep 08 17:19:56.046358 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.046295    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" event={"ID":"0bd319fe-5c42-46c0-9b6c-037abac0f8a6","Type":"ContainerDied","Data":"0e01010820413ed7578f35c23cf0d2aacd42a72b695a656aa30c9f5be65dbc11"}
Sep 08 17:19:56.046404 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.046375    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6" event={"ID":"0bd319fe-5c42-46c0-9b6c-037abac0f8a6","Type":"ContainerDied","Data":"a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7"}
Sep 08 17:19:56.046444 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.046402    2542 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2823bdeb14077db9c5f2901157c5a0bba616227ef8abf5598c49604f95247a7"
Sep 08 17:19:56.050791 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.050773    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:19:56.053717 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.053700    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:19:56.186187 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186152    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t5pv\" (UniqueName: \"kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv\") pod \"903344ac-7a1a-44e1-901d-4eeec854c8cf\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") "
Sep 08 17:19:56.186429 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186199    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume\") pod \"903344ac-7a1a-44e1-901d-4eeec854c8cf\" (UID: \"903344ac-7a1a-44e1-901d-4eeec854c8cf\") "
Sep 08 17:19:56.186429 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186236    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhfwk\" (UniqueName: \"kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk\") pod \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") "
Sep 08 17:19:56.186429 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186261    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume\") pod \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\" (UID: \"0bd319fe-5c42-46c0-9b6c-037abac0f8a6\") "
Sep 08 17:19:56.186672 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186647    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "903344ac-7a1a-44e1-901d-4eeec854c8cf" (UID: "903344ac-7a1a-44e1-901d-4eeec854c8cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:19:56.186728 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.186697    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bd319fe-5c42-46c0-9b6c-037abac0f8a6" (UID: "0bd319fe-5c42-46c0-9b6c-037abac0f8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:19:56.188399 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.188380    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk" (OuterVolumeSpecName: "kube-api-access-lhfwk") pod "0bd319fe-5c42-46c0-9b6c-037abac0f8a6" (UID: "0bd319fe-5c42-46c0-9b6c-037abac0f8a6"). InnerVolumeSpecName "kube-api-access-lhfwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:19:56.188460 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.188432    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv" (OuterVolumeSpecName: "kube-api-access-2t5pv") pod "903344ac-7a1a-44e1-901d-4eeec854c8cf" (UID: "903344ac-7a1a-44e1-901d-4eeec854c8cf"). InnerVolumeSpecName "kube-api-access-2t5pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:19:56.287774 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.287678    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5pv\" (UniqueName: \"kubernetes.io/projected/903344ac-7a1a-44e1-901d-4eeec854c8cf-kube-api-access-2t5pv\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:56.287774 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.287712    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903344ac-7a1a-44e1-901d-4eeec854c8cf-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:56.287774 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.287722    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfwk\" (UniqueName: \"kubernetes.io/projected/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-kube-api-access-lhfwk\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:56.287774 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:56.287730    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd319fe-5c42-46c0-9b6c-037abac0f8a6-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:19:57.051073 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.051039    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"
Sep 08 17:19:57.051551 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.051041    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"
Sep 08 17:19:57.075694 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.075664    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:19:57.084415 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.084383    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-c05c9f-6d5974dc94-wvds6"]
Sep 08 17:19:57.098074 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.098037    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:19:57.102469 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.102446    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-c05c9f-7f8d6d74ff-wc4qb"]
Sep 08 17:19:57.116620 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.116595    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" path="/var/lib/kubelet/pods/0bd319fe-5c42-46c0-9b6c-037abac0f8a6/volumes"
Sep 08 17:19:57.116870 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.116858    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9279ee-05a7-4e4e-b63e-9853b8165e38" path="/var/lib/kubelet/pods/0e9279ee-05a7-4e4e-b63e-9853b8165e38/volumes"
Sep 08 17:19:57.117131 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.117119    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903344ac-7a1a-44e1-901d-4eeec854c8cf" path="/var/lib/kubelet/pods/903344ac-7a1a-44e1-901d-4eeec854c8cf/volumes"
Sep 08 17:19:57.232369 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.232311    2542 scope.go:117] "RemoveContainer" containerID="ee1f9981c8c376ef3cc7b8b30827876635b0f9b13db1b6cee64e3ac5cf9dec29"
Sep 08 17:19:57.240882 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:19:57.240863    2542 scope.go:117] "RemoveContainer" containerID="0e01010820413ed7578f35c23cf0d2aacd42a72b695a656aa30c9f5be65dbc11"
Sep 08 17:20:13.701147 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.701112    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:20:13.701804 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.701361    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" podUID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" containerName="payload-processing" containerID="cri-o://169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8" gracePeriod=30
Sep 08 17:20:13.719104 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.719041    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:20:13.719342 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.719297    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" podUID="6267107c-8b73-49c3-93ca-6cd1da0a5831" containerName="payload-pre-processing" containerID="cri-o://0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9" gracePeriod=30
Sep 08 17:20:13.965114 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.965049    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:20:13.996392 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:13.996368    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:20:14.137821 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.137790    2542 generic.go:356] "Generic (PLEG): container finished" podID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" containerID="169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8" exitCode=0
Sep 08 17:20:14.138006 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.137835    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"
Sep 08 17:20:14.138006 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.137874    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" event={"ID":"a35e73bb-2be3-48ab-a4e4-27b908b237e9","Type":"ContainerDied","Data":"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"}
Sep 08 17:20:14.138006 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.137929    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4" event={"ID":"a35e73bb-2be3-48ab-a4e4-27b908b237e9","Type":"ContainerDied","Data":"ce265b5704b394d1e25ff1140f0a7e69ac8a68a64c905ef83f20f49c33fed92c"}
Sep 08 17:20:14.138006 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.137957    2542 scope.go:117] "RemoveContainer" containerID="169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"
Sep 08 17:20:14.139235 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.139210    2542 generic.go:356] "Generic (PLEG): container finished" podID="6267107c-8b73-49c3-93ca-6cd1da0a5831" containerID="0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9" exitCode=0
Sep 08 17:20:14.139343 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.139250    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"
Sep 08 17:20:14.139343 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.139268    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" event={"ID":"6267107c-8b73-49c3-93ca-6cd1da0a5831","Type":"ContainerDied","Data":"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"}
Sep 08 17:20:14.139343 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.139310    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw" event={"ID":"6267107c-8b73-49c3-93ca-6cd1da0a5831","Type":"ContainerDied","Data":"0c285e8757c13846c10d661c5fffe01eeecf7bb01c700beab685ccf95958013a"}
Sep 08 17:20:14.143643 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.143619    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume\") pod \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") "
Sep 08 17:20:14.143729 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.143657    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrr2b\" (UniqueName: \"kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b\") pod \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\" (UID: \"a35e73bb-2be3-48ab-a4e4-27b908b237e9\") "
Sep 08 17:20:14.143729 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.143682    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkq2h\" (UniqueName: \"kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h\") pod \"6267107c-8b73-49c3-93ca-6cd1da0a5831\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") "
Sep 08 17:20:14.143910 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.143787    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume\") pod \"6267107c-8b73-49c3-93ca-6cd1da0a5831\" (UID: \"6267107c-8b73-49c3-93ca-6cd1da0a5831\") "
Sep 08 17:20:14.144071 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.144044    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a35e73bb-2be3-48ab-a4e4-27b908b237e9" (UID: "a35e73bb-2be3-48ab-a4e4-27b908b237e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:20:14.144180 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.144103    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6267107c-8b73-49c3-93ca-6cd1da0a5831" (UID: "6267107c-8b73-49c3-93ca-6cd1da0a5831"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 17:20:14.145845 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.145821    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h" (OuterVolumeSpecName: "kube-api-access-jkq2h") pod "6267107c-8b73-49c3-93ca-6cd1da0a5831" (UID: "6267107c-8b73-49c3-93ca-6cd1da0a5831"). InnerVolumeSpecName "kube-api-access-jkq2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:20:14.145933 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.145898    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b" (OuterVolumeSpecName: "kube-api-access-lrr2b") pod "a35e73bb-2be3-48ab-a4e4-27b908b237e9" (UID: "a35e73bb-2be3-48ab-a4e4-27b908b237e9"). InnerVolumeSpecName "kube-api-access-lrr2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:20:14.146705 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.146689    2542 scope.go:117] "RemoveContainer" containerID="169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"
Sep 08 17:20:14.146983 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:20:14.146964    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8\": container with ID starting with 169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8 not found: ID does not exist" containerID="169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"
Sep 08 17:20:14.147042 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.146988    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8"} err="failed to get container status \"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8\": rpc error: code = NotFound desc = could not find container \"169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8\": container with ID starting with 169dbeb86c9e791ac2f48fbd4211f18f73e6486767f3aeadb126ac28ad27e8d8 not found: ID does not exist"
Sep 08 17:20:14.147042 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.147004    2542 scope.go:117] "RemoveContainer" containerID="0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"
Sep 08 17:20:14.154632 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.154613    2542 scope.go:117] "RemoveContainer" containerID="0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"
Sep 08 17:20:14.154924 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:20:14.154896    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9\": container with ID starting with 0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9 not found: ID does not exist" containerID="0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"
Sep 08 17:20:14.155011 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.154929    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9"} err="failed to get container status \"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9\": rpc error: code = NotFound desc = could not find container \"0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9\": container with ID starting with 0af3bf88ca26e89308558062a2084a874765878c9a38f70547fa4a56be8890a9 not found: ID does not exist"
Sep 08 17:20:14.245283 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.245193    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6267107c-8b73-49c3-93ca-6cd1da0a5831-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:20:14.245283 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.245222    2542 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a35e73bb-2be3-48ab-a4e4-27b908b237e9-plugins-config-volume\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:20:14.245283 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.245231    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrr2b\" (UniqueName: \"kubernetes.io/projected/a35e73bb-2be3-48ab-a4e4-27b908b237e9-kube-api-access-lrr2b\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:20:14.245283 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.245241    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkq2h\" (UniqueName: \"kubernetes.io/projected/6267107c-8b73-49c3-93ca-6cd1da0a5831-kube-api-access-jkq2h\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:20:14.469617 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.469583    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:20:14.472472 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.472446    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-338600-595cc8d7b9-892f4"]
Sep 08 17:20:14.483193 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.483169    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:20:14.487280 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:14.487258    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-338600-5898d4f7db-xh8dw"]
Sep 08 17:20:15.116994 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:15.116913    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6267107c-8b73-49c3-93ca-6cd1da0a5831" path="/var/lib/kubelet/pods/6267107c-8b73-49c3-93ca-6cd1da0a5831/volumes"
Sep 08 17:20:15.117373 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:15.117270    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" path="/var/lib/kubelet/pods/a35e73bb-2be3-48ab-a4e4-27b908b237e9/volumes"
Sep 08 17:20:57.076227 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:57.076195    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:20:57.078984 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:57.078963    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:20:57.273304 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:57.273271    2542 scope.go:117] "RemoveContainer" containerID="427fc78fc0c14207e45268d737c8e9fe43e085fb844cf4b1664896602c7776fa"
Sep 08 17:20:57.281208 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:20:57.281183    2542 scope.go:117] "RemoveContainer" containerID="1e58b7d7a7800450fa0c19881259cc0e06cc41d7aa29a068aca2b265deea4e0e"
Sep 08 17:21:57.310066 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:21:57.310029    2542 scope.go:117] "RemoveContainer" containerID="dd84c69f14b998407529d0c120d3bbbbba0a590aaee4f30aa60522d777f135a0"
Sep 08 17:24:57.191366 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191314    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:24:57.191804 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191768    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" containerName="payload-pre-processing"
Sep 08 17:24:57.191804 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191782    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" containerName="payload-pre-processing"
Sep 08 17:24:57.191804 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191801    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6267107c-8b73-49c3-93ca-6cd1da0a5831" containerName="payload-pre-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191811    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="6267107c-8b73-49c3-93ca-6cd1da0a5831" containerName="payload-pre-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191822    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e000fd5-b6cc-46da-957b-eaa144df1860" containerName="payload-pre-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191831    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e000fd5-b6cc-46da-957b-eaa144df1860" containerName="payload-pre-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191848    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903344ac-7a1a-44e1-901d-4eeec854c8cf" containerName="payload-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191858    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="903344ac-7a1a-44e1-901d-4eeec854c8cf" containerName="payload-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191868    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" containerName="revoke-keys"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191877    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" containerName="revoke-keys"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191891    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" containerName="payload-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191900    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" containerName="payload-processing"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191916    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e2c179-fe20-4766-aca5-1ef879040f60" containerName="main"
Sep 08 17:24:57.191926 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191926    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e2c179-fe20-4766-aca5-1ef879040f60" containerName="main"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191940    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" containerName="payload-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.191949    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" containerName="payload-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192016    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="903344ac-7a1a-44e1-901d-4eeec854c8cf" containerName="payload-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192026    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e000fd5-b6cc-46da-957b-eaa144df1860" containerName="payload-pre-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192036    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35e73bb-2be3-48ab-a4e4-27b908b237e9" containerName="payload-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192047    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2cc50a-22fa-4c03-83bc-1e256e1ce56d" containerName="revoke-keys"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192056    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd319fe-5c42-46c0-9b6c-037abac0f8a6" containerName="payload-pre-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192065    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="6267107c-8b73-49c3-93ca-6cd1da0a5831" containerName="payload-pre-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192074    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b2f8f2-18c4-4e39-ab9a-28102e2e6730" containerName="payload-processing"
Sep 08 17:24:57.192261 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.192081    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e2c179-fe20-4766-aca5-1ef879040f60" containerName="main"
Sep 08 17:24:57.195187 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.195168    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.197822 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.197798    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8cd6fb80-kserve-self-signed-certs\""
Sep 08 17:24:57.208108 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.208084    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:24:57.247630 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247597    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.247807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247644    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.247807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247671    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.247807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247743    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.247807 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247781    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.247947 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.247827    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppt8\" (UniqueName: \"kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.341096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.341064    2542 scope.go:117] "RemoveContainer" containerID="cdd715068d26c0a562f17e3e8f78626775c8732183085a4a1326158c2fbf09d3"
Sep 08 17:24:57.348538 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348511    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348656 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348573    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppt8\" (UniqueName: \"kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348656 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348599    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348656 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348627    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348795 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348686    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348795 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348739    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.348980 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348939    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.349097 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.348978    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.349225 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.349142    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.349502 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.349471    2542 scope.go:117] "RemoveContainer" containerID="2c1ca63c7e0bde858505b98462ca80e7b999c7cce10e2a571e5a216e8aebb3e9"
Sep 08 17:24:57.351027 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.351004    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.351397 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.351377    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.356736 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.356713    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppt8\" (UniqueName: \"kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8\") pod \"e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") " pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.504558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.504473    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:24:57.778227 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:57.778139    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:24:57.781014 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:24:57.780985    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a9b30b_7dae_40ad_9a25_44ce160ba0b4.slice/crio-5248105017b8e7b99d7ec24e93332f486c460cf94197e7ac9ad4d4199abf303c WatchSource:0}: Error finding container 5248105017b8e7b99d7ec24e93332f486c460cf94197e7ac9ad4d4199abf303c: Status 404 returned error can't find the container with id 5248105017b8e7b99d7ec24e93332f486c460cf94197e7ac9ad4d4199abf303c
Sep 08 17:24:58.387661 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:58.387628    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" event={"ID":"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4","Type":"ContainerStarted","Data":"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"}
Sep 08 17:24:58.387661 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:58.387666    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" event={"ID":"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4","Type":"ContainerStarted","Data":"5248105017b8e7b99d7ec24e93332f486c460cf94197e7ac9ad4d4199abf303c"}
Sep 08 17:24:58.405024 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:24:58.404973    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" podStartSLOduration=1.4049560300000001 podStartE2EDuration="1.40495603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:24:58.403554372 +0000 UTC m=+2041.870760570" watchObservedRunningTime="2026-09-08 17:24:58.40495603 +0000 UTC m=+2041.872162223"
Sep 08 17:25:07.505506 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:07.505451    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:07.505506 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:07.505513    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:07.517984 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:07.517958    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:08.441202 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:08.441164    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:09.409171 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.409133    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:09.412487 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.412469    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.415252 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.415227    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8cd6fb80-kserve-self-signed-certs\""
Sep 08 17:25:09.423350 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.423313    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:09.455017 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.454983    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.455017 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.455015    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.455228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.455034    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s5j\" (UniqueName: \"kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.455228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.455083    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.455228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.455141    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.455228 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.455196    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.555982 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.555943    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.555982 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.555978    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556259 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556009    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s5j\" (UniqueName: \"kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556259 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556035    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556259 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556247    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556518 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556501    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556504    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556560 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556538    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.556625 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.556587    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.558463 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.558434    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.558903 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.558881    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.564616 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.564596    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s5j\" (UniqueName: \"kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j\") pod \"e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") " pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.721866 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.721831    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:09.992225 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:09.992199    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:09.994717 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:25:09.994690    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b877fae_0025_4cfa_b9ea_8f91de05f9ea.slice/crio-2ae48bd65ae4d21755697229401338590b31e81a91cc91a7680e5bd522acfe32 WatchSource:0}: Error finding container 2ae48bd65ae4d21755697229401338590b31e81a91cc91a7680e5bd522acfe32: Status 404 returned error can't find the container with id 2ae48bd65ae4d21755697229401338590b31e81a91cc91a7680e5bd522acfe32
Sep 08 17:25:10.442654 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:10.442620    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" event={"ID":"9b877fae-0025-4cfa-b9ea-8f91de05f9ea","Type":"ContainerStarted","Data":"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"}
Sep 08 17:25:10.442654 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:10.442660    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" event={"ID":"9b877fae-0025-4cfa-b9ea-8f91de05f9ea","Type":"ContainerStarted","Data":"2ae48bd65ae4d21755697229401338590b31e81a91cc91a7680e5bd522acfe32"}
Sep 08 17:25:10.462423 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:10.462374    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" podStartSLOduration=1.462356652 podStartE2EDuration="1.462356652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 17:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:25:10.460880219 +0000 UTC m=+2053.928086412" watchObservedRunningTime="2026-09-08 17:25:10.462356652 +0000 UTC m=+2053.929562845"
Sep 08 17:25:19.723016 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:19.722919    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:19.723466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:19.723118    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:19.735671 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:19.735640    2542 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:20.498450 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:20.498414    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:37.107875 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:37.107838    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:25:39.276910 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:39.276875    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:52.127873 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.127828    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" podUID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" containerName="main" containerID="cri-o://0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf" gracePeriod=45
Sep 08 17:25:52.374003 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.373976    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:52.431028 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431002    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431033    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431053    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431067    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431090    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431205 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431126    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ppt8\" (UniqueName: \"kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8\") pod \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\" (UID: \"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4\") "
Sep 08 17:25:52.431499 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431281    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:52.431499 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431303    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:52.431499 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431380    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.431499 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.431379    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home" (OuterVolumeSpecName: "home") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:52.433209 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.433179    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8" (OuterVolumeSpecName: "kube-api-access-9ppt8") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "kube-api-access-9ppt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:25:52.433209 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.433200    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:25:52.433371 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.433205    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm" (OuterVolumeSpecName: "dshm") pod "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" (UID: "a0a9b30b-7dae-40ad-9a25-44ce160ba0b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:52.532558 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.532524    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.532715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.532572    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.532715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.532582    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.532715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.532590    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.532715 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.532598    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppt8\" (UniqueName: \"kubernetes.io/projected/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4-kube-api-access-9ppt8\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:52.629102 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.629062    2542 generic.go:356] "Generic (PLEG): container finished" podID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" containerID="0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf" exitCode=0
Sep 08 17:25:52.629264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.629109    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" event={"ID":"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4","Type":"ContainerDied","Data":"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"}
Sep 08 17:25:52.629264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.629126    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"
Sep 08 17:25:52.629264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.629148    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk" event={"ID":"a0a9b30b-7dae-40ad-9a25-44ce160ba0b4","Type":"ContainerDied","Data":"5248105017b8e7b99d7ec24e93332f486c460cf94197e7ac9ad4d4199abf303c"}
Sep 08 17:25:52.629264 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.629170    2542 scope.go:117] "RemoveContainer" containerID="0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"
Sep 08 17:25:52.639929 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.639906    2542 scope.go:117] "RemoveContainer" containerID="0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"
Sep 08 17:25:52.640187 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:25:52.640163    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf\": container with ID starting with 0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf not found: ID does not exist" containerID="0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"
Sep 08 17:25:52.640243 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.640198    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf"} err="failed to get container status \"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf\": rpc error: code = NotFound desc = could not find container \"0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf\": container with ID starting with 0796de2c7bbd34379eb920ee5c0614e073580dbec0b7c6904e1805bb5a7045cf not found: ID does not exist"
Sep 08 17:25:52.650053 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.650025    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:25:52.652305 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:52.652284    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8cd6fb80-kserve-67b5947b79-dmvpk"]
Sep 08 17:25:53.116159 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:53.116124    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" path="/var/lib/kubelet/pods/a0a9b30b-7dae-40ad-9a25-44ce160ba0b4/volumes"
Sep 08 17:25:54.297015 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.296965    2542 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" podUID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" containerName="main" containerID="cri-o://d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a" gracePeriod=45
Sep 08 17:25:54.547266 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.547207    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:54.640668 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.640630    2542 generic.go:356] "Generic (PLEG): container finished" podID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" containerID="d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a" exitCode=0
Sep 08 17:25:54.640838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.640671    2542 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"
Sep 08 17:25:54.640838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.640715    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" event={"ID":"9b877fae-0025-4cfa-b9ea-8f91de05f9ea","Type":"ContainerDied","Data":"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"}
Sep 08 17:25:54.640838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.640756    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65" event={"ID":"9b877fae-0025-4cfa-b9ea-8f91de05f9ea","Type":"ContainerDied","Data":"2ae48bd65ae4d21755697229401338590b31e81a91cc91a7680e5bd522acfe32"}
Sep 08 17:25:54.640838 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.640776    2542 scope.go:117] "RemoveContainer" containerID="d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"
Sep 08 17:25:54.648907 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.648889    2542 scope.go:117] "RemoveContainer" containerID="d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"
Sep 08 17:25:54.649179 ip-10-0-133-106 kubenswrapper[2542]: E0908 17:25:54.649160    2542 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a\": container with ID starting with d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a not found: ID does not exist" containerID="d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"
Sep 08 17:25:54.649243 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.649183    2542 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a"} err="failed to get container status \"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a\": rpc error: code = NotFound desc = could not find container \"d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a\": container with ID starting with d5c00492d8a84ddc6170a3f7550fdcca1085f75ff1db9706a12cab66f62c810a not found: ID does not exist"
Sep 08 17:25:54.651479 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651458    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651595 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651506    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6s5j\" (UniqueName: \"kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651595 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651536    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651681 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651604    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651681 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651624    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651681 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651651    2542 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home\") pod \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\" (UID: \"9b877fae-0025-4cfa-b9ea-8f91de05f9ea\") "
Sep 08 17:25:54.651849 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651726    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:54.651908 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651860    2542 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tmp-dir\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.651966 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651924    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:54.651966 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.651945    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home" (OuterVolumeSpecName: "home") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:54.653632 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.653598    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j" (OuterVolumeSpecName: "kube-api-access-k6s5j") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "kube-api-access-k6s5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 17:25:54.653725 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.653654    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 17:25:54.653725 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.653668    2542 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm" (OuterVolumeSpecName: "dshm") pod "9b877fae-0025-4cfa-b9ea-8f91de05f9ea" (UID: "9b877fae-0025-4cfa-b9ea-8f91de05f9ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 17:25:54.752679 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.752648    2542 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-model-cache\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.752679 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.752677    2542 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-home\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.752679 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.752687    2542 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s5j\" (UniqueName: \"kubernetes.io/projected/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-kube-api-access-k6s5j\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.752969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.752696    2542 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-tls-certs\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.752969 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.752705    2542 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b877fae-0025-4cfa-b9ea-8f91de05f9ea-dshm\") on node \"ip-10-0-133-106.ec2.internal\" DevicePath \"\""
Sep 08 17:25:54.963096 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.963059    2542 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:54.969121 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:54.969097    2542 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8cd6fb80-kserve-5c75958ff5-ccl65"]
Sep 08 17:25:55.116332 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:55.116287    2542 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" path="/var/lib/kubelet/pods/9b877fae-0025-4cfa-b9ea-8f91de05f9ea/volumes"
Sep 08 17:25:57.107733 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:57.107702    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:25:57.110318 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:25:57.110296    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2j8w7_da26b116-98f9-4269-afb0-d8c118b580a0/ovn-acl-logging/0.log"
Sep 08 17:26:37.952170 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:37.952139    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-b28bb_95295e73-58df-4462-82be-e0f2617357fc/server/0.log"
Sep 08 17:26:38.066030 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:38.065998    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-68hh5_fd0a5bf0-7fe5-47aa-b848-73c84cc71f79/manager/0.log"
Sep 08 17:26:38.295789 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:38.295711    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-lxtjp_7ec0db90-7fd4-43d9-99e0-714138427a4a/manager/0.log"
Sep 08 17:26:38.301717 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:38.301699    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-lxtjp_7ec0db90-7fd4-43d9-99e0-714138427a4a/copy-manifests/0.log"
Sep 08 17:26:38.988068 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:38.988035    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814780-8fd9g_bd8a28ca-81a5-450b-bab2-9f5041a4cb23/cleanup/0.log"
Sep 08 17:26:39.091611 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:39.091582    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814795-9g5xv_d04b5bc8-5a6b-4ce0-8389-e57ff03a1507/cleanup/0.log"
Sep 08 17:26:41.325301 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:41.325269    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-vslwz_83282811-8ca5-4e9d-93f8-03277ff5d0c8/kube-auth-proxy/0.log"
Sep 08 17:26:41.557376 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:41.557342    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-vww8c_d338767d-21e5-4f2e-aa1f-95beee4ad7c0/payload-pre-processing/0.log"
Sep 08 17:26:42.177304 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:42.177274    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qzc6g_90dcba84-5f18-49b2-bad6-511f49565d3c/main/0.log"
Sep 08 17:26:42.397672 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:42.397634    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-65t98_a1cbc8f2-ceeb-4867-8054-dc2ea3a5d9a4/main/0.log"
Sep 08 17:26:42.604710 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:42.604631    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff6hqq_432ee6b2-5c1b-4df4-be17-81a23d82cd90/main/0.log"
Sep 08 17:26:42.829603 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:42.829576    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wmqkz_3551ab1c-99b9-491a-a4b1-6735290cc656/main/0.log"
Sep 08 17:26:50.911881 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:50.911807    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fp7hm_8c813616-6cba-4cb3-8d91-22703afd28c0/global-pull-secret-syncer/0.log"
Sep 08 17:26:51.123505 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:51.123472    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jj7qm_e04be129-fe5a-47c5-a4cc-969300cc8f85/konnectivity-agent/0.log"
Sep 08 17:26:51.165342 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:51.165262    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-106.ec2.internal_984ceb6c332935eca9a7c33929be3059/haproxy/0.log"
Sep 08 17:26:57.893457 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:57.893433    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnhg8_2a483673-11bf-44ec-9286-4a6c6a483a39/node-exporter/0.log"
Sep 08 17:26:57.916768 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:57.916730    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnhg8_2a483673-11bf-44ec-9286-4a6c6a483a39/kube-rbac-proxy/0.log"
Sep 08 17:26:57.938095 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:57.938067    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qnhg8_2a483673-11bf-44ec-9286-4a6c6a483a39/init-textfile/0.log"
Sep 08 17:26:59.993268 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:26:59.993240    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-cwjkc_6478c3cb-d09f-4aa6-bad7-5c87a320b03e/networking-console-plugin/0.log"
Sep 08 17:27:01.167655 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.167618    2542 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"]
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.167932    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" containerName="main"
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.167942    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" containerName="main"
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.167953    2542 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" containerName="main"
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.167958    2542 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" containerName="main"
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.168009    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a9b30b-7dae-40ad-9a25-44ce160ba0b4" containerName="main"
Sep 08 17:27:01.168038 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.168016    2542 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b877fae-0025-4cfa-b9ea-8f91de05f9ea" containerName="main"
Sep 08 17:27:01.170978 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.170957    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.173466 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.173440    2542 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7djdj\"/\"default-dockercfg-q2jfr\""
Sep 08 17:27:01.173581 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.173496    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"kube-root-ca.crt\""
Sep 08 17:27:01.174410 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.174391    2542 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7djdj\"/\"openshift-service-ca.crt\""
Sep 08 17:27:01.179442 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.179423    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"]
Sep 08 17:27:01.312636 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.312605    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-proc\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.312636 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.312637    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-lib-modules\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.312840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.312668    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mxs\" (UniqueName: \"kubernetes.io/projected/b3271eef-2ed6-4b24-819c-2f07d1d9943a-kube-api-access-95mxs\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.312840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.312715    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-sys\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.312840 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.312756    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-podres\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413567 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413526    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-podres\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413744 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413691    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-proc\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413744 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413711    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-lib-modules\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413744 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413727    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-podres\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413852 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413731    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mxs\" (UniqueName: \"kubernetes.io/projected/b3271eef-2ed6-4b24-819c-2f07d1d9943a-kube-api-access-95mxs\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413852 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413807    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-proc\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413852 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413812    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-sys\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413952 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413883    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-sys\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.413952 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.413896    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3271eef-2ed6-4b24-819c-2f07d1d9943a-lib-modules\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.421994 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.421926    2542 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mxs\" (UniqueName: \"kubernetes.io/projected/b3271eef-2ed6-4b24-819c-2f07d1d9943a-kube-api-access-95mxs\") pod \"perf-node-gather-daemonset-2f7ds\" (UID: \"b3271eef-2ed6-4b24-819c-2f07d1d9943a\") " pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.480964 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.480931    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.751454 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.751419    2542 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"]
Sep 08 17:27:01.754937 ip-10-0-133-106 kubenswrapper[2542]: W0908 17:27:01.754908    2542 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3271eef_2ed6_4b24_819c_2f07d1d9943a.slice/crio-3a4ecc861427d271205f9d37bd0d1b3086091cf5db78102c18534fb0d0334ec8 WatchSource:0}: Error finding container 3a4ecc861427d271205f9d37bd0d1b3086091cf5db78102c18534fb0d0334ec8: Status 404 returned error can't find the container with id 3a4ecc861427d271205f9d37bd0d1b3086091cf5db78102c18534fb0d0334ec8
Sep 08 17:27:01.937308 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.937272    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds" event={"ID":"b3271eef-2ed6-4b24-819c-2f07d1d9943a","Type":"ContainerStarted","Data":"dc7fa29c442ed58b64a4501d403a0befe25dfb5751dd3637b9c74760e4d20f97"}
Sep 08 17:27:01.937524 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.937343    2542 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds" event={"ID":"b3271eef-2ed6-4b24-819c-2f07d1d9943a","Type":"ContainerStarted","Data":"3a4ecc861427d271205f9d37bd0d1b3086091cf5db78102c18534fb0d0334ec8"}
Sep 08 17:27:01.937524 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.937370    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:01.953313 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:01.953254    2542 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds" podStartSLOduration=0.953239398 podStartE2EDuration="953.239398ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 17:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 17:27:01.95158656 +0000 UTC m=+2165.418792747" watchObservedRunningTime="2026-09-08 17:27:01.953239398 +0000 UTC m=+2165.420445591"
Sep 08 17:27:03.046428 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:03.046394    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hvpz_30ac2086-1fcf-4ec7-9ed2-8202a55eb804/dns/0.log"
Sep 08 17:27:03.067174 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:03.067146    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hvpz_30ac2086-1fcf-4ec7-9ed2-8202a55eb804/kube-rbac-proxy/0.log"
Sep 08 17:27:03.171508 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:03.171476    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rnlsr_1ca4fe31-13cd-4ee8-b7df-a200e5f77c3f/dns-node-resolver/0.log"
Sep 08 17:27:03.686663 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:03.686624    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d6ffbb4b-zxzkk_c5807b69-c876-46b5-84c8-1ac5e5336805/registry/0.log"
Sep 08 17:27:03.729146 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:03.729112    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxpmq_798f9bdd-3b9f-408d-9bf2-cf639dcd2de6/node-ca/0.log"
Sep 08 17:27:05.058628 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:05.058591    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-vslwz_83282811-8ca5-4e9d-93f8-03277ff5d0c8/kube-auth-proxy/0.log"
Sep 08 17:27:05.136540 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:05.136482    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-vww8c_d338767d-21e5-4f2e-aa1f-95beee4ad7c0/payload-pre-processing/0.log"
Sep 08 17:27:06.040772 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:06.040737    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jxlfm_fa00cca0-57a4-4301-9b8d-e9d09393ba5f/serve-healthcheck-canary/0.log"
Sep 08 17:27:06.624119 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:06.624092    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99cv_36c70a95-9295-4368-a484-1c6dacf4d6e5/kube-rbac-proxy/0.log"
Sep 08 17:27:06.645953 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:06.645930    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99cv_36c70a95-9295-4368-a484-1c6dacf4d6e5/exporter/0.log"
Sep 08 17:27:06.665943 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:06.665919    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k99cv_36c70a95-9295-4368-a484-1c6dacf4d6e5/extractor/0.log"
Sep 08 17:27:07.951418 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:07.951389    2542 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7djdj/perf-node-gather-daemonset-2f7ds"
Sep 08 17:27:09.229551 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:09.229523    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-b28bb_95295e73-58df-4462-82be-e0f2617357fc/server/0.log"
Sep 08 17:27:09.305861 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:09.305835    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-68hh5_fd0a5bf0-7fe5-47aa-b848-73c84cc71f79/manager/0.log"
Sep 08 17:27:09.423484 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:09.423451    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-lxtjp_7ec0db90-7fd4-43d9-99e0-714138427a4a/manager/0.log"
Sep 08 17:27:09.442353 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:09.442332    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84f58c54b-lxtjp_7ec0db90-7fd4-43d9-99e0-714138427a4a/copy-manifests/0.log"
Sep 08 17:27:10.980705 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:10.980636    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74fc5c54f8-jgz68_34357644-0983-4721-8004-653ce1db3b05/manager/0.log"
Sep 08 17:27:12.040294 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:12.040269    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-vslwz_83282811-8ca5-4e9d-93f8-03277ff5d0c8/kube-auth-proxy/0.log"
Sep 08 17:27:12.101776 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:12.101750    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-vww8c_d338767d-21e5-4f2e-aa1f-95beee4ad7c0/payload-pre-processing/0.log"
Sep 08 17:27:14.119908 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:14.119865    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d8677cc5f-vslwz_83282811-8ca5-4e9d-93f8-03277ff5d0c8/kube-auth-proxy/0.log"
Sep 08 17:27:14.184118 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:14.184085    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-vww8c_d338767d-21e5-4f2e-aa1f-95beee4ad7c0/payload-pre-processing/0.log"
Sep 08 17:27:16.063812 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:16.063781    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-7lkqs_3e8ecea5-0f73-48cc-b24c-c297127a12f1/migrator/0.log"
Sep 08 17:27:16.082521 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:16.082493    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-7lkqs_3e8ecea5-0f73-48cc-b24c-c297127a12f1/graceful-termination/0.log"
Sep 08 17:27:17.440909 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.440883    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/kube-multus-additional-cni-plugins/0.log"
Sep 08 17:27:17.460668 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.460642    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/egress-router-binary-copy/0.log"
Sep 08 17:27:17.483133 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.483108    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/cni-plugins/0.log"
Sep 08 17:27:17.502882 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.502857    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/bond-cni-plugin/0.log"
Sep 08 17:27:17.522230 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.522202    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/routeoverride-cni/0.log"
Sep 08 17:27:17.542140 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.542109    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/whereabouts-cni-bincopy/0.log"
Sep 08 17:27:17.562814 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.562792    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4gcjd_4c9cf456-163c-4f9b-ae20-b7f5ce64c673/whereabouts-cni/0.log"
Sep 08 17:27:17.905448 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.905368    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wr7bb_9cb35f70-fad2-4215-a1c0-0f4be01c68ec/kube-multus/0.log"
Sep 08 17:27:17.925222 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.925191    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2rxnz_bdd679a3-1c44-40bc-9c88-f97be23eec57/network-metrics-daemon/0.log"
Sep 08 17:27:17.942422 ip-10-0-133-106 kubenswrapper[2542]: I0908 17:27:17.942398    2542 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2rxnz_bdd679a3-1c44-40bc-9c88-f97be23eec57/kube-rbac-proxy/0.log"