Sep 06 13:57:42.844102 ip-10-0-133-52 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 13:57:43.293916 ip-10-0-133-52 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 13:57:43.293916 ip-10-0-133-52 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 13:57:43.293916 ip-10-0-133-52 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 13:57:43.293916 ip-10-0-133-52 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 13:57:43.293916 ip-10-0-133-52 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 13:57:43.296133 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.296040    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 13:57:43.300125 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300102    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.300125 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300119    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.300125 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300123    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.300125 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300127    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.300125 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300132    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300137    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300141    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300145    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300150    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300154    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300162    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300168    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300174    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300178    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300183    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300187    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300192    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300196    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300200    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300204    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300208    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300220    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300224    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.300422 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300228    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300232    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300236    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300239    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300243    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300247    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300251    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300255    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300258    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300262    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300267    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300271    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300275    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300279    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300283    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300288    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300293    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300297    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300302    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300306    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.301227 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300311    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300315    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300320    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300324    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300328    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300332    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300337    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300340    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300345    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300349    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300353    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300361    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300367    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300372    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300376    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300380    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300384    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300388    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300393    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.301969 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300398    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300402    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300406    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300410    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300414    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300418    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300423    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300428    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300432    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300445    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300450    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300454    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300458    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300462    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300466    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300470    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300475    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300479    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300483    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300487    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.302518 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300491    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300496    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300500    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300504    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300509    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300514    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300517    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300524    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300529    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300533    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300537    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300542    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300546    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300552    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300559    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300563    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.300568    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301789    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301800    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.303073 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301805    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301810    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301857    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301862    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301866    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301869    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301873    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301876    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301878    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301881    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301884    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301888    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301891    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301894    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301896    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301900    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301903    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301906    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301909    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301911    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.303541 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301914    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301916    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301919    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301922    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301924    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301928    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301932    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301936    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301940    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301943    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301946    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301948    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301951    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301954    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301958    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301961    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301964    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301967    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301970    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301972    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.304061 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301975    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301978    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301981    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301984    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301986    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301989    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301991    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301994    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.301997    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302000    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302004    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302008    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302010    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302013    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302016    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302019    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302022    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302025    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302027    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.304554 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302030    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302032    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302035    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302038    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302040    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302043    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302046    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302050    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302053    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302057    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302059    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302063    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302065    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302068    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302070    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302073    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302076    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302078    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302081    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302083    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.305048 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302088    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302091    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302093    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302096    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302099    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302101    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302104    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302107    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302109    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302112    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302114    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302117    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302121    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302124    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302126    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302129    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302132    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302134    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302533    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.305538 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302539    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302543    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302547    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302550    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302553    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302556    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302559    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302561    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302564    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302567    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302569    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302572    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302574    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302577    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302579    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302582    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302585    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302587    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302590    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302592    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302595    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.306018 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302597    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302600    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302602    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302605    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302608    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302611    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302614    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302616    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302619    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302621    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302624    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302627    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302630    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302633    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302637    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302641    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302644    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302647    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302649    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.306525 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302652    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302655    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302658    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302660    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302663    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302665    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302668    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302670    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302673    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302675    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302678    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302681    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302684    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302686    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302689    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302691    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302694    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302697    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302700    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302703    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.307004 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302705    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302708    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302711    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302713    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302716    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302719    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302721    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302724    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302727    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302729    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302731    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302734    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302736    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302739    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302742    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302745    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302748    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302750    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302753    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.307536 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302756    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302776    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302780    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302784    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302788    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302790    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302793    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302796    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302798    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302801    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302804    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302806    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302809    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302812    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302814    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302816    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302819    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302822    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.302825    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302910    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302951    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 13:57:43.308027 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302961    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302966    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302971    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302975    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302979    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302984    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302987    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302990    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302994    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.302997    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303018    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303022    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303026    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303029    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303032    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303035    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303039    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303042    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303045    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303048    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303051    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303056    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303059    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 13:57:43.308541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303065    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303069    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303072    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303075    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303079    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303082    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303085    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303089    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303092    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303095    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303098    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303102    2576 flags.go:64] FLAG: --enable-server="true"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303105    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303109    2576 flags.go:64] FLAG: --event-burst="100"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303112    2576 flags.go:64] FLAG: --event-qps="50"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303115    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303119    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303122    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303125    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303128    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303132    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303135    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303138    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303140    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303143    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 13:57:43.309153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303146    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303149    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303152    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303155    2576 flags.go:64] FLAG: --feature-gates=""
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303159    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303162    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303165    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303170    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303173    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303176    2576 flags.go:64] FLAG: --help="false"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303179    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303182    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303185    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303188    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303191    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303195    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303198    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303201    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303204    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303208    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303213    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303217    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303220    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303222    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 13:57:43.309754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303225    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303228    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303231    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303234    2576 flags.go:64] FLAG: --lock-file=""
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303236    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303239    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303243    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303248    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303251    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303254    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303257    2576 flags.go:64] FLAG: --logging-format="text"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303260    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303263    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303266    2576 flags.go:64] FLAG: --manifest-url=""
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303269    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303275    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303278    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303283    2576 flags.go:64] FLAG: --max-pods="110"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303286    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303289    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303292    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303295    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303298    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303301    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303304    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 13:57:43.310367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303312    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303315    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303317    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303323    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303326    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303332    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303335    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303338    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303341    2576 flags.go:64] FLAG: --port="10250"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303344    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303347    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bccfbb5a8c947e58"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303350    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303353    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303356    2576 flags.go:64] FLAG: --register-node="true"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303359    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303363    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303365    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303368    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303371    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303375    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303378    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303382    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303385    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303390    2576 flags.go:64] FLAG: --runonce="false"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303393    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 13:57:43.311037 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303396    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303399    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303402    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303404    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303408    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303411    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303414    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303417    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303420    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303423    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303427    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303431    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303433    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303436    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303442    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303446    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303449    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303452    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303455    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303458    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303461    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303465    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303468    2576 flags.go:64] FLAG: --v="2"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303472    2576 flags.go:64] FLAG: --version="false"
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303476    2576 flags.go:64] FLAG: --vmodule=""
Sep 06 13:57:43.311638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303480    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.303483    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303597    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303601    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303605    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303608    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303611    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303614    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303616    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303619    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303622    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303624    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303627    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303629    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303632    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303634    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303637    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303641    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303644    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303647    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.312269 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303649    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303652    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303654    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303657    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303659    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303662    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303664    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303667    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303669    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303672    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303674    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303677    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303680    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303683    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303686    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303689    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303693    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303696    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303699    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.312788 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303702    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303705    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303708    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303711    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303713    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303716    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303719    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303722    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303725    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303727    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303732    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303735    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303738    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303741    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303743    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303746    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303748    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303751    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303754    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.313250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303756    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303777    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303783    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303789    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303792    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303795    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303798    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303801    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303803    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303806    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303809    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303811    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303814    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303817    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303820    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303822    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303825    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303827    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303830    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303832    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.313705 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303835    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303837    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303840    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303844    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303847    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303849    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303852    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303861    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303863    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303866    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303870    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303874    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303878    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303881    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303885    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303887    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303890    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303893    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303896    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303898    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.314297 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303901    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303904    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.303906    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.304721    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.312904    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.312926    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.313868    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.313876    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313943    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313949    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313953    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313956    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313959    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313962    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313965    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.314885 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313968    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313970    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313973    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313976    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313979    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313981    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313984    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313986    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313990    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313994    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.313997    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314000    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314002    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314005    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314008    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314010    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314012    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314015    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314018    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.315257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314020    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314023    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314026    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314029    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314038    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314041    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314044    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314046    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314048    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314051    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314053    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314056    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314059    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314062    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314066    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314069    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314072    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314076    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314080    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.315730 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314083    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314086    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314089    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314092    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314094    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314097    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314099    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314103    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314105    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314108    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314110    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314112    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314115    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314117    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314120    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314123    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314125    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314128    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314137    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314140    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.316207 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314142    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314145    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314147    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314150    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314152    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314155    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314157    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314160    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314162    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314165    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314167    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314169    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314172    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314174    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314177    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314179    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314182    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314184    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314187    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314189    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.316690 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314192    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314195    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314197    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314200    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314202    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314205    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314207    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314210    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314213    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314215    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314218    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314226    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314229    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314231    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.314236    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 13:57:43.317190 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314393    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314397    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314400    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314403    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314405    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314408    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314410    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314413    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314415    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314418    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314420    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314423    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314426    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314428    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314431    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314433    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314436    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314438    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314441    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314443    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 13:57:43.317566 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314446    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314448    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314451    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314453    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314456    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314458    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314461    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314464    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314466    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314475    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314478    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314480    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314483    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314485    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314488    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314490    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314493    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314495    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314498    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314500    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 13:57:43.318079 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314503    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314505    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314508    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314510    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314513    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314516    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314518    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314521    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314523    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314526    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314528    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314531    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314533    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314536    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314538    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314542    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314545    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314548    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314551    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314554    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 13:57:43.318574 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314557    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314559    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314567    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314570    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314572    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314575    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314578    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314580    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314582    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314585    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314587    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314590    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314592    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314595    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314597    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314600    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314602    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314605    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314607    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314609    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 13:57:43.319090 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314612    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314616    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314620    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314623    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314628    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314631    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314634    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314636    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314639    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314642    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314645    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314648    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314651    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314654    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314656    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314665    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314668    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314670    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 13:57:43.319578 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:43.314673    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.314678    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.314817    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.317409    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.318561    2576 server.go:1019] "Starting client certificate rotation"
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.318681    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 13:57:43.320028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.318712    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 13:57:43.321478 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.321462    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 13:57:43.327803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.327746    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 13:57:43.340474 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.340451    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 06 13:57:43.344146 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.344125    2576 log.go:25] "Validated CRI v1 image API"
Sep 06 13:57:43.345131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.345103    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 13:57:43.350510 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.350477    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 963263bc-520b-4546-a0b4-f5ab05d2308c:/dev/nvme0n1p3 ff0896c1-30e8-488d-946b-768c2982c52d:/dev/nvme0n1p4]
Sep 06 13:57:43.350510 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.350503    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 13:57:43.355379 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.355361    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 13:57:43.355895 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.355784    2576 manager.go:216] Machine: {Timestamp:2026-09-06 13:57:43.35430387 +0000 UTC m=+0.380369057 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101805 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d27c047c8893d77b600701cfd82d5 SystemUUID:ec2d27c0-47c8-893d-77b6-00701cfd82d5 BootID:22216481-52ad-4ebd-aee5-b2a2a2a5a4e5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5c:14:c1:0b:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5c:14:c1:0b:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:b6:ac:df:53:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 13:57:43.355895 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.355891    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 13:57:43.355991 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.355983    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 13:57:43.359739 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.359710    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 13:57:43.359933 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.359744    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 13:57:43.361433 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.361419    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 13:57:43.361501 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.361438    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 13:57:43.361501 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.361451    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 13:57:43.362315 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.362303    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 13:57:43.364199 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.364187    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 13:57:43.368792 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.368777    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 13:57:43.368940 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.368928    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 13:57:43.368997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.368967    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 13:57:43.368997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.368982    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 13:57:43.369091 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.369002    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 13:57:43.369091 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.369019    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 06 13:57:43.369091 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.369033    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 13:57:43.369488 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.369474    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 13:57:43.369554 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.369498    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 13:57:43.372182 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.372159    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 13:57:43.375092 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375077    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 13:57:43.375136 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375108    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 13:57:43.375271 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375261    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 13:57:43.375308 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375279    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 13:57:43.375308 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375285    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 13:57:43.375308 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375290    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 13:57:43.375308 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375297    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 13:57:43.375308 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375302    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 13:57:43.375698 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375677    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pk4gg"
Sep 06 13:57:43.375998 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375987    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 13:57:43.376030 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.375998    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 13:57:43.376030 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.376006    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 13:57:43.376030 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.376012    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 13:57:43.376030 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.376021    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 13:57:43.376030 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.376032    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 13:57:43.377911 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.377900    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 13:57:43.377911 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.377910    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 13:57:43.382629 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.382585    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pk4gg"
Sep 06 13:57:43.383728 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.383707    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 13:57:43.383835 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.383818    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 13:57:43.383890 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.383875    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 13:57:43.385953 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.385936    2576 server.go:1268] "Started kubelet"
Sep 06 13:57:43.386053 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.386016    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 13:57:43.386153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.386102    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 13:57:43.386203 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.386181    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 13:57:43.387050 ip-10-0-133-52 systemd[1]: Started Kubernetes Kubelet.
Sep 06 13:57:43.387795 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.387737    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 13:57:43.388114 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.388095    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 13:57:43.390608 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.390432    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 13:57:43.394132 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.392966    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-52.ec2.internal.18d2bff58dc50d47  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-52.ec2.internal,UID:ip-10-0-133-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-52.ec2.internal,},FirstTimestamp:2026-09-06 13:57:43.385902407 +0000 UTC m=+0.411967574,LastTimestamp:2026-09-06 13:57:43.385902407 +0000 UTC m=+0.411967574,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-52.ec2.internal,}"
Sep 06 13:57:43.528855 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.528825    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 13:57:43.528855 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.528849    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 13:57:43.529064 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.529043    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:43.529711 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529691    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 13:57:43.529827 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529696    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 13:57:43.529827 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529739    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 13:57:43.529934 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529831    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 13:57:43.529934 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529841    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 13:57:43.529934 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529868    2576 factory.go:55] Registering systemd factory
Sep 06 13:57:43.529934 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.529885    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 06 13:57:43.530137 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530121    2576 factory.go:153] Registering CRI-O factory
Sep 06 13:57:43.530137 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530140    2576 factory.go:223] Registration of the crio container factory successfully
Sep 06 13:57:43.530280 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530198    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 13:57:43.530280 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530220    2576 factory.go:103] Registering Raw factory
Sep 06 13:57:43.530280 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530235    2576 manager.go:1196] Started watching for new ooms in manager
Sep 06 13:57:43.530653 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.530637    2576 manager.go:349] Starting recovery of all containers
Sep 06 13:57:43.531311 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.531286    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 06 13:57:43.531427 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.531313    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 06 13:57:43.537640 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.537613    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 13:57:43.542817 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.542799    2576 manager.go:354] Recovery completed
Sep 06 13:57:43.548205 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.548143    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.549964 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.549949    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.550028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.549975    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.550028 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.549986    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.550575 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.550560    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 13:57:43.550575 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.550571    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 13:57:43.550696 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.550584    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.550696 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.550589    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 13:57:43.551267 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.551246    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.551331 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.551269    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.551331 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.551279    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.552858 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.552847    2576 policy_none.go:49] "None policy: Start"
Sep 06 13:57:43.552900 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.552862    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 13:57:43.552900 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.552871    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 13:57:43.554732 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.554722    2576 policy_none.go:47] "Start"
Sep 06 13:57:43.597403 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597386    2576 manager.go:337] "Starting Device Plugin manager"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.597425    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597435    2576 server.go:87] "Starting device plugin registration server"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597715    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597729    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597862    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597963    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.597972    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.598480    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.598550    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.603570    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.603618    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.603639    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.603679    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 13:57:43.606217 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.605248    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:43.698693 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.698660    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.699692 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.699672    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.699779 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.699714    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.699779 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.699725    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.699779 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.699748    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.703953 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.703931    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"]
Sep 06 13:57:43.704023 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.704012    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.705060 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.705043    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.705121 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.705066    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-52.ec2.internal\": node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:43.705424 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.705410    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.705470 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.705437    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.705470 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.705447    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.707801 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.707789    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.707858 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.707847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.707908 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.707872    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.708530 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708512    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.708631 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708541    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.708631 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708555    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.708631 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708513    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.708631 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708606    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.708631 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.708616    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.711061 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.711041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.711187 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.711078    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 13:57:43.712348 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.712333    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 13:57:43.712438 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.712356    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 13:57:43.712438 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.712366    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeHasSufficientPID"
Sep 06 13:57:43.719871 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.719852    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:43.731523 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.731493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c714293c914577f6a860a52a7e5762b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.ec2.internal\" (UID: \"c714293c914577f6a860a52a7e5762b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.731638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.731532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.731638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.731559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.748771 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.748736    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-52.ec2.internal\" not found" node="ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.752244 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.752228    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-52.ec2.internal\" not found" node="ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.820870 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.820797    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:43.831831 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c714293c914577f6a860a52a7e5762b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.ec2.internal\" (UID: \"c714293c914577f6a860a52a7e5762b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.831890 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.831890 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.831956 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c714293c914577f6a860a52a7e5762b6-config\") pod \"kube-apiserver-proxy-ip-10-0-133-52.ec2.internal\" (UID: \"c714293c914577f6a860a52a7e5762b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.831956 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.832015 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:43.831941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b704edf06adae0a8b77a9b0792edab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal\" (UID: \"77b704edf06adae0a8b77a9b0792edab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:43.921274 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:43.921242    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.021849 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.021812    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.053337 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.053295    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:44.056649 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.056617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:44.123099 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.123002    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.225513 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.225470    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.261229 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.261185    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:44.318646 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.318594    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 13:57:44.319173 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.318811    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 13:57:44.319173 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.318814    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 13:57:44.326325 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.326301    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.405960 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.405879    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:44.427015 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.426991    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.527979 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:44.527933    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-52.ec2.internal\" not found"
Sep 06 13:57:44.529177 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.529127    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 13:52:43 +0000 UTC" deadline="2028-04-15 21:48:07.010313724 +0000 UTC"
Sep 06 13:57:44.529269 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.529177    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14095h50m22.4811407s"
Sep 06 13:57:44.543855 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.543832    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:44.629068 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.629035    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:44.639093 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.639065    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 13:57:44.640215 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.640198    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal"
Sep 06 13:57:44.649383 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.649362    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 13:57:44.691851 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:44.691808    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc714293c914577f6a860a52a7e5762b6.slice/crio-0499c75bd3381f168738cd10c63c60d9d52973d4a995a5fe659872c83e53a00a WatchSource:0}: Error finding container 0499c75bd3381f168738cd10c63c60d9d52973d4a995a5fe659872c83e53a00a: Status 404 returned error can't find the container with id 0499c75bd3381f168738cd10c63c60d9d52973d4a995a5fe659872c83e53a00a
Sep 06 13:57:44.692121 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:44.692100    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b704edf06adae0a8b77a9b0792edab.slice/crio-c56e7010edcb4be49fc4d6b6aa5dd29b02f83726f6363273f8664e1f2217c614 WatchSource:0}: Error finding container c56e7010edcb4be49fc4d6b6aa5dd29b02f83726f6363273f8664e1f2217c614: Status 404 returned error can't find the container with id c56e7010edcb4be49fc4d6b6aa5dd29b02f83726f6363273f8664e1f2217c614
Sep 06 13:57:44.697743 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:44.697728    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 13:57:45.327908 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.327873    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:45.371941 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.371899    2576 apiserver.go:52] "Watching apiserver"
Sep 06 13:57:45.378229 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.378190    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 13:57:45.378641 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.378606    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-l95xv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal","openshift-multus/multus-additional-cni-plugins-f4xxx","openshift-multus/network-metrics-daemon-ldcbb","openshift-network-diagnostics/network-check-target-lzmtl","kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz","openshift-image-registry/node-ca-dzd9g","openshift-multus/multus-hhjln","openshift-network-operator/iptables-alerter-qbhlm","openshift-ovn-kubernetes/ovnkube-node-ntgbk","kube-system/global-pull-secret-syncer-nmr22","kube-system/konnectivity-agent-nsfxb","openshift-cluster-node-tuning-operator/tuned-xdtzv"]
Sep 06 13:57:45.381831 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.381804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.384742 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.384164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.384742 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.384538    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-drm4s\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.385611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.385865    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.386144    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.386698    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w2tnm\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.387004    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.387307    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 13:57:45.387678 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.387656    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 13:57:45.389275 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.388926    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 13:57:45.389275 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.388975    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.389275 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.389153    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.389512 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.389450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.391896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.391990    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392370    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392564    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392594    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392600    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fnwx9\""
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392767    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 13:57:45.394364 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.392801    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 13:57:45.394856 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.394391    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:45.394856 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.394474    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:45.397144 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.396842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.399363 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.399084    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.399476 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.399383    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fg5mg\""
Sep 06 13:57:45.400021 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.399695    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.400021 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.399712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.401956 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.401927    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 13:57:45.402672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.402225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.402672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.402379    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml9b7\""
Sep 06 13:57:45.404354 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.404332    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.406141 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.405476    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tbflc\""
Sep 06 13:57:45.406141 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.405679    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.406141 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.405746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.406141 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.405683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 13:57:45.408180 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.407722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.408180 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.407957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cxmpn\""
Sep 06 13:57:45.408326 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.408317    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.408567 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.408547    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.410774 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.410215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bzfm2\""
Sep 06 13:57:45.410774 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.410547    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 13:57:45.410935 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.410779    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 13:57:45.411163 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.411134    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 13:57:45.411410 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.411390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.411496 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.411472    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:45.412979 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.412958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.415617 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.415598    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 13:57:45.415960 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.415941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v6h5m\""
Sep 06 13:57:45.416055 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.415963    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 13:57:45.431315 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.431292    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-registration-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-netns\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440600    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-hostroot\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440677    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-kubernetes\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-slash\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.440756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.441509 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-node-log\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.441509 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-conf\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.441509 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-var-lib-kubelet\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.441509 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-etc-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.441509 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.440888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-script-lib\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-system-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe20f3c5-a2b7-488d-af61-b5845188ff63-serviceca\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd75l\" (UniqueName: \"kubernetes.io/projected/fe20f3c5-a2b7-488d-af61-b5845188ff63-kube-api-access-jd75l\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441787    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cdp\" (UniqueName: \"kubernetes.io/projected/7fe351ac-3d3e-4845-986b-6179ba1e60b7-kube-api-access-j4cdp\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-kubelet-config\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-device-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.441897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-env-overrides\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-bin\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.441992    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-netns\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c6p8\" (UniqueName: \"kubernetes.io/projected/bae7c569-ded6-42e5-9ee9-4c499a27b95b-kube-api-access-2c6p8\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cnibin\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-lib-modules\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-log-socket\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-etc-kubernetes\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-modprobe-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysconfig\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-systemd\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-sys\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.442313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7s4\" (UniqueName: \"kubernetes.io/projected/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-kube-api-access-5x7s4\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe351ac-3d3e-4845-986b-6179ba1e60b7-host-slash\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cni-binary-copy\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-multus\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6de74b58-56f8-4cb2-9823-7f4034db8d3d-agent-certs\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442553    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnf6\" (UniqueName: \"kubernetes.io/projected/079a9bd2-3476-4b43-8f7f-25eaa03baabf-kube-api-access-djnf6\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-multus-certs\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-tmp\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-socket-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndj4\" (UniqueName: \"kubernetes.io/projected/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kube-api-access-jndj4\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-kubelet\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-daemon-config\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bae7c569-ded6-42e5-9ee9-4c499a27b95b-hosts-file\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.444374 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6de74b58-56f8-4cb2-9823-7f4034db8d3d-konnectivity-ca\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2f25\" (UniqueName: \"kubernetes.io/projected/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-kube-api-access-b2f25\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.442974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cnibin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9cz2\" (UniqueName: \"kubernetes.io/projected/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-kube-api-access-w9cz2\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe20f3c5-a2b7-488d-af61-b5845188ff63-host\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjk2h\" (UniqueName: \"kubernetes.io/projected/2f0bd173-2c38-4019-9a81-84cb8910cd0c-kube-api-access-tjk2h\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-run\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-kubelet\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-systemd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-bin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-system-cni-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-dbus\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-sys-fs\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.445278 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-socket-dir-parent\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-var-lib-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-conf-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-systemd-units\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-ovn\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-netd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-k8s-cni-cncf-io\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae7c569-ded6-42e5-9ee9-4c499a27b95b-tmp-dir\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-host\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-config\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-tuned\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.446160 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.443837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe351ac-3d3e-4845-986b-6179ba1e60b7-iptables-alerter-script\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.529909 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.529871    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 13:57:45.539522 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.539480    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 13:57:45.544506 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-systemd\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-sys\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7s4\" (UniqueName: \"kubernetes.io/projected/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-kube-api-access-5x7s4\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe351ac-3d3e-4845-986b-6179ba1e60b7-host-slash\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cni-binary-copy\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-multus\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6de74b58-56f8-4cb2-9823-7f4034db8d3d-agent-certs\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.544672 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnf6\" (UniqueName: \"kubernetes.io/projected/079a9bd2-3476-4b43-8f7f-25eaa03baabf-kube-api-access-djnf6\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-multus-certs\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-tmp\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-socket-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndj4\" (UniqueName: \"kubernetes.io/projected/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kube-api-access-jndj4\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-kubelet\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-daemon-config\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bae7c569-ded6-42e5-9ee9-4c499a27b95b-hosts-file\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.544997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6de74b58-56f8-4cb2-9823-7f4034db8d3d-konnectivity-ca\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.545067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545055    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2f25\" (UniqueName: \"kubernetes.io/projected/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-kube-api-access-b2f25\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cnibin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545166    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545253    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-kubelet\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-systemd\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-sys\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9cz2\" (UniqueName: \"kubernetes.io/projected/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-kube-api-access-w9cz2\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe20f3c5-a2b7-488d-af61-b5845188ff63-host\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjk2h\" (UniqueName: \"kubernetes.io/projected/2f0bd173-2c38-4019-9a81-84cb8910cd0c-kube-api-access-tjk2h\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-run\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-multus-certs\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-kubelet\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-systemd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.545804 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.545634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-bin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.546608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6de74b58-56f8-4cb2-9823-7f4034db8d3d-konnectivity-ca\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.546608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bae7c569-ded6-42e5-9ee9-4c499a27b95b-hosts-file\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.546608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546288    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 13:57:45.546608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.546608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-system-cni-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546608    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-dbus\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-sys-fs\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-socket-dir-parent\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-var-lib-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-daemon-config\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-conf-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.546887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-systemd-units\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-ovn\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-netd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-k8s-cni-cncf-io\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.546987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae7c569-ded6-42e5-9ee9-4c499a27b95b-tmp-dir\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-host\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-config\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cnibin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-tuned\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.547148    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe351ac-3d3e-4845-986b-6179ba1e60b7-iptables-alerter-script\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-registration-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.547228    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:46.047190116 +0000 UTC m=+3.073255283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-netns\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547267    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-registration-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.547463 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-hostroot\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-kubernetes\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-slash\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-node-log\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-conf\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-var-lib-kubelet\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-etc-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-script-lib\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-system-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe20f3c5-a2b7-488d-af61-b5845188ff63-serviceca\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd75l\" (UniqueName: \"kubernetes.io/projected/fe20f3c5-a2b7-488d-af61-b5845188ff63-kube-api-access-jd75l\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cdp\" (UniqueName: \"kubernetes.io/projected/7fe351ac-3d3e-4845-986b-6179ba1e60b7-kube-api-access-j4cdp\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-kubelet-config\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-binary-copy\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.548425 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-device-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-env-overrides\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe20f3c5-a2b7-488d-af61-b5845188ff63-host\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547950    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-bin\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.547993    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-bin\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-netns\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c6p8\" (UniqueName: \"kubernetes.io/projected/bae7c569-ded6-42e5-9ee9-4c499a27b95b-kube-api-access-2c6p8\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-multus\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cnibin\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-lib-modules\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-log-socket\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-etc-kubernetes\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-run\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-modprobe-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548219    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysconfig\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-kubelet\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.548970 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae7c569-ded6-42e5-9ee9-4c499a27b95b-tmp-dir\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548329    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-dbus\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-cni-binary-copy\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-socket-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-host\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7fe351ac-3d3e-4845-986b-6179ba1e60b7-host-slash\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548421    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-k8s-cni-cncf-io\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-var-lib-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-log-socket\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-conf-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-sys-fs\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-systemd-units\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.548568    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-ovn\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.548629    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:46.048610777 +0000 UTC m=+3.074675957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-cni-netd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.549736 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-run-netns\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-run-netns\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-etc-kubernetes\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-config\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-modprobe-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0a218de8-5016-4c86-971d-3f14763198c1-kubelet-config\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-device-dir\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-etc-selinux\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovnkube-script-lib\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-systemd\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysconfig\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.548501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-var-lib-kubelet\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-system-cni-dir\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-system-cni-dir\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-d\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-hostroot\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.550644 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079a9bd2-3476-4b43-8f7f-25eaa03baabf-env-overrides\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549730    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-kubernetes\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-host-var-lib-cni-bin\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-host-slash\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-node-log\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-run-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-sysctl-conf\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7fe351ac-3d3e-4845-986b-6179ba1e60b7-iptables-alerter-script\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.549996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-lib-modules\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.550021    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-multus-socket-dir-parent\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.550295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe20f3c5-a2b7-488d-af61-b5845188ff63-serviceca\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.550356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079a9bd2-3476-4b43-8f7f-25eaa03baabf-etc-openvswitch\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.551518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.550400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2f0bd173-2c38-4019-9a81-84cb8910cd0c-cnibin\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.552210 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.551839    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:57:45.552210 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.552045    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:57:45.552210 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.552061    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:45.552210 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:45.552130    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:57:46.052111246 +0000 UTC m=+3.078176407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:45.553218 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.553191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-tmp\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.553559 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.553538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6de74b58-56f8-4cb2-9823-7f4034db8d3d-agent-certs\") pod \"konnectivity-agent-nsfxb\" (UID: \"6de74b58-56f8-4cb2-9823-7f4034db8d3d\") " pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.553825 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.553778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-etc-tuned\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.556802 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.555914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079a9bd2-3476-4b43-8f7f-25eaa03baabf-ovn-node-metrics-cert\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.557148 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.557100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7s4\" (UniqueName: \"kubernetes.io/projected/ed6ca2cf-4bc8-4cdf-a430-00968c77f40a-kube-api-access-5x7s4\") pod \"tuned-xdtzv\" (UID: \"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a\") " pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.559019 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.558284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnf6\" (UniqueName: \"kubernetes.io/projected/079a9bd2-3476-4b43-8f7f-25eaa03baabf-kube-api-access-djnf6\") pod \"ovnkube-node-ntgbk\" (UID: \"079a9bd2-3476-4b43-8f7f-25eaa03baabf\") " pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.559019 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.558547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2f25\" (UniqueName: \"kubernetes.io/projected/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-kube-api-access-b2f25\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:45.563095 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.563073    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjk2h\" (UniqueName: \"kubernetes.io/projected/2f0bd173-2c38-4019-9a81-84cb8910cd0c-kube-api-access-tjk2h\") pod \"multus-additional-cni-plugins-f4xxx\" (UID: \"2f0bd173-2c38-4019-9a81-84cb8910cd0c\") " pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.565059 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.565032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9cz2\" (UniqueName: \"kubernetes.io/projected/c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a-kube-api-access-w9cz2\") pod \"multus-hhjln\" (UID: \"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a\") " pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.567421 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.567395    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-26rzg"
Sep 06 13:57:45.569362 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.569336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cdp\" (UniqueName: \"kubernetes.io/projected/7fe351ac-3d3e-4845-986b-6179ba1e60b7-kube-api-access-j4cdp\") pod \"iptables-alerter-qbhlm\" (UID: \"7fe351ac-3d3e-4845-986b-6179ba1e60b7\") " pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.570973 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.570909    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c6p8\" (UniqueName: \"kubernetes.io/projected/bae7c569-ded6-42e5-9ee9-4c499a27b95b-kube-api-access-2c6p8\") pod \"node-resolver-l95xv\" (UID: \"bae7c569-ded6-42e5-9ee9-4c499a27b95b\") " pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.572179 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.572126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndj4\" (UniqueName: \"kubernetes.io/projected/48cdb294-0609-4b78-bea2-cd3ff0fbbcf7-kube-api-access-jndj4\") pod \"aws-ebs-csi-driver-node-mk8sz\" (UID: \"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.572594 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.572544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd75l\" (UniqueName: \"kubernetes.io/projected/fe20f3c5-a2b7-488d-af61-b5845188ff63-kube-api-access-jd75l\") pod \"node-ca-dzd9g\" (UID: \"fe20f3c5-a2b7-488d-af61-b5845188ff63\") " pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.574571 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.574544    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-26rzg"
Sep 06 13:57:45.609673 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.609557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal" event={"ID":"77b704edf06adae0a8b77a9b0792edab","Type":"ContainerStarted","Data":"c56e7010edcb4be49fc4d6b6aa5dd29b02f83726f6363273f8664e1f2217c614"}
Sep 06 13:57:45.611224 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.611190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal" event={"ID":"c714293c914577f6a860a52a7e5762b6","Type":"ContainerStarted","Data":"0499c75bd3381f168738cd10c63c60d9d52973d4a995a5fe659872c83e53a00a"}
Sep 06 13:57:45.705264 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.705214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz"
Sep 06 13:57:45.715154 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.715122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:57:45.718215 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.718172    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cdb294_0609_4b78_bea2_cd3ff0fbbcf7.slice/crio-3a4679b5062ff6864f2d67a06bc93de9b73fd9784c4e091b61292a2ed7f1e44e WatchSource:0}: Error finding container 3a4679b5062ff6864f2d67a06bc93de9b73fd9784c4e091b61292a2ed7f1e44e: Status 404 returned error can't find the container with id 3a4679b5062ff6864f2d67a06bc93de9b73fd9784c4e091b61292a2ed7f1e44e
Sep 06 13:57:45.726399 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.726372    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079a9bd2_3476_4b43_8f7f_25eaa03baabf.slice/crio-210a94b43f372cdbb8ff674a9995e3a20e23791e7a9e2885dace1adccf512fcc WatchSource:0}: Error finding container 210a94b43f372cdbb8ff674a9995e3a20e23791e7a9e2885dace1adccf512fcc: Status 404 returned error can't find the container with id 210a94b43f372cdbb8ff674a9995e3a20e23791e7a9e2885dace1adccf512fcc
Sep 06 13:57:45.730942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.730864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f4xxx"
Sep 06 13:57:45.734093 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.734036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdtzv"
Sep 06 13:57:45.743899 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.743868    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0bd173_2c38_4019_9a81_84cb8910cd0c.slice/crio-68df6f3ae6640991b661e4695e12beed72bb9eeb39d38b42fa7025a4740d15eb WatchSource:0}: Error finding container 68df6f3ae6640991b661e4695e12beed72bb9eeb39d38b42fa7025a4740d15eb: Status 404 returned error can't find the container with id 68df6f3ae6640991b661e4695e12beed72bb9eeb39d38b42fa7025a4740d15eb
Sep 06 13:57:45.747000 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.746910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhjln"
Sep 06 13:57:45.747344 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.747317    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6ca2cf_4bc8_4cdf_a430_00968c77f40a.slice/crio-3a9a8d6174053d2240da304e37e72c719e69f2be5cac35f2972d0f8e95b1582d WatchSource:0}: Error finding container 3a9a8d6174053d2240da304e37e72c719e69f2be5cac35f2972d0f8e95b1582d: Status 404 returned error can't find the container with id 3a9a8d6174053d2240da304e37e72c719e69f2be5cac35f2972d0f8e95b1582d
Sep 06 13:57:45.755285 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.755123    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dzd9g"
Sep 06 13:57:45.758257 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.758122    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9733cc8_58c9_4ca6_beb0_69dcf9b0a04a.slice/crio-c1919e0c86f2a0ee87177e90ee838274fd3b6909d2dad8f23848ac1a087a4276 WatchSource:0}: Error finding container c1919e0c86f2a0ee87177e90ee838274fd3b6909d2dad8f23848ac1a087a4276: Status 404 returned error can't find the container with id c1919e0c86f2a0ee87177e90ee838274fd3b6909d2dad8f23848ac1a087a4276
Sep 06 13:57:45.764748 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.764722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l95xv"
Sep 06 13:57:45.766000 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.765970    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe20f3c5_a2b7_488d_af61_b5845188ff63.slice/crio-d991207e5cae59e6e91379f13bff536c1411b65bc828ea98288dd3488768ea96 WatchSource:0}: Error finding container d991207e5cae59e6e91379f13bff536c1411b65bc828ea98288dd3488768ea96: Status 404 returned error can't find the container with id d991207e5cae59e6e91379f13bff536c1411b65bc828ea98288dd3488768ea96
Sep 06 13:57:45.774015 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.773992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qbhlm"
Sep 06 13:57:45.777738 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.777702    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae7c569_ded6_42e5_9ee9_4c499a27b95b.slice/crio-4c98cb30d227914278277c8855db7a1f129ca411aff4536cfc9fbc9c8627fb50 WatchSource:0}: Error finding container 4c98cb30d227914278277c8855db7a1f129ca411aff4536cfc9fbc9c8627fb50: Status 404 returned error can't find the container with id 4c98cb30d227914278277c8855db7a1f129ca411aff4536cfc9fbc9c8627fb50
Sep 06 13:57:45.781251 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.781234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:57:45.785250 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:57:45.785219    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe351ac_3d3e_4845_986b_6179ba1e60b7.slice/crio-bceb1c4e422ff306a0467bb51a2e583c5712cfbb84f5b5a5fa44e9724e1cf904 WatchSource:0}: Error finding container bceb1c4e422ff306a0467bb51a2e583c5712cfbb84f5b5a5fa44e9724e1cf904: Status 404 returned error can't find the container with id bceb1c4e422ff306a0467bb51a2e583c5712cfbb84f5b5a5fa44e9724e1cf904
Sep 06 13:57:45.987003 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:45.986972    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 13:57:46.052312 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.052269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:46.052467 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.052330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:46.052467 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.052364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:46.052467 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052435    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:46.052467 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052462    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052516    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052537    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052546    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:47.052500625 +0000 UTC m=+4.078565799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052594    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:47.052577082 +0000 UTC m=+4.078642249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:46.052677 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.052616    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:57:47.052605847 +0000 UTC m=+4.078671009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:46.580134 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.580023    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:45 +0000 UTC" deadline="2028-04-21 18:14:38.863937646 +0000 UTC"
Sep 06 13:57:46.580134 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.580076    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14236h16m52.283866584s"
Sep 06 13:57:46.605265 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.604514    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:46.605265 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.604651    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:46.605265 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.605107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:46.605265 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:46.605212    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:46.615141 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.615104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerStarted","Data":"68df6f3ae6640991b661e4695e12beed72bb9eeb39d38b42fa7025a4740d15eb"}
Sep 06 13:57:46.620105 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.620073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"210a94b43f372cdbb8ff674a9995e3a20e23791e7a9e2885dace1adccf512fcc"}
Sep 06 13:57:46.627421 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.627389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz" event={"ID":"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7","Type":"ContainerStarted","Data":"3a4679b5062ff6864f2d67a06bc93de9b73fd9784c4e091b61292a2ed7f1e44e"}
Sep 06 13:57:46.629252 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.629176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qbhlm" event={"ID":"7fe351ac-3d3e-4845-986b-6179ba1e60b7","Type":"ContainerStarted","Data":"bceb1c4e422ff306a0467bb51a2e583c5712cfbb84f5b5a5fa44e9724e1cf904"}
Sep 06 13:57:46.630888 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.630856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l95xv" event={"ID":"bae7c569-ded6-42e5-9ee9-4c499a27b95b","Type":"ContainerStarted","Data":"4c98cb30d227914278277c8855db7a1f129ca411aff4536cfc9fbc9c8627fb50"}
Sep 06 13:57:46.634001 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.633975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzd9g" event={"ID":"fe20f3c5-a2b7-488d-af61-b5845188ff63","Type":"ContainerStarted","Data":"d991207e5cae59e6e91379f13bff536c1411b65bc828ea98288dd3488768ea96"}
Sep 06 13:57:46.635469 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.635392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhjln" event={"ID":"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a","Type":"ContainerStarted","Data":"c1919e0c86f2a0ee87177e90ee838274fd3b6909d2dad8f23848ac1a087a4276"}
Sep 06 13:57:46.637328 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:46.637303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdtzv" event={"ID":"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a","Type":"ContainerStarted","Data":"3a9a8d6174053d2240da304e37e72c719e69f2be5cac35f2972d0f8e95b1582d"}
Sep 06 13:57:47.060416 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.060193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:47.060416 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.060246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:47.060416 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.060306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:47.060715 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.060449    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:57:47.060715 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.060471    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:57:47.060715 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.060484    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:47.060715 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.060544    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:57:49.060525215 +0000 UTC m=+6.086590370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:47.061020 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.060971    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:47.061077 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.061021    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:49.061006159 +0000 UTC m=+6.087071314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:47.061137 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.061077    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:47.061137 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.061110    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:49.061098412 +0000 UTC m=+6.087163571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:47.581255 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.581194    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 13:52:45 +0000 UTC" deadline="2028-06-11 20:43:51.582557376 +0000 UTC"
Sep 06 13:57:47.581255 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.581237    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15462h46m4.001324978s"
Sep 06 13:57:47.606000 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.605965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:47.606156 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:47.606103    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:47.650014 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:47.649932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nsfxb" event={"ID":"6de74b58-56f8-4cb2-9823-7f4034db8d3d","Type":"ContainerStarted","Data":"661dbe1a968332d5d720ec31fdf6d37a42fa4462f15301e63c0e8bac4d299a03"}
Sep 06 13:57:48.604856 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:48.604133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:48.604856 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:48.604258    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:48.604856 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:48.604656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:48.604856 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:48.604785    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:48.665807 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:48.665351    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal" event={"ID":"77b704edf06adae0a8b77a9b0792edab","Type":"ContainerStarted","Data":"e38fe8620d36b4953dfba2c44bb4d50a3048e85d40a343f3dcf5672d307d2ae1"}
Sep 06 13:57:48.685520 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:48.684955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal" event={"ID":"c714293c914577f6a860a52a7e5762b6","Type":"ContainerStarted","Data":"e200fc6d7c3b53b66b4ccb3603cfb916ce51979a58e0b8f43b89384f796bb0ae"}
Sep 06 13:57:49.077665 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:49.077622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:49.077864 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:49.077700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:49.077864 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:49.077738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:49.077991 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.077922    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:49.077991 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.077986    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.077966662 +0000 UTC m=+10.104031819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:49.078448 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078393    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:57:49.078448 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078417    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:57:49.078448 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078432    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:49.078631 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078476    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.0784619 +0000 UTC m=+10.104527054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:49.078631 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078534    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:49.078631 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.078565    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:57:53.078555585 +0000 UTC m=+10.104620744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:49.606928 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:49.606895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:49.607337 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:49.607033    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:50.604146 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:50.604104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:50.604336 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:50.604236    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:50.604794 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:50.604459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:50.604794 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:50.604556    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:50.692751 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:50.692705    2576 generic.go:356] "Generic (PLEG): container finished" podID="77b704edf06adae0a8b77a9b0792edab" containerID="e38fe8620d36b4953dfba2c44bb4d50a3048e85d40a343f3dcf5672d307d2ae1" exitCode=0
Sep 06 13:57:50.693219 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:50.692777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal" event={"ID":"77b704edf06adae0a8b77a9b0792edab","Type":"ContainerDied","Data":"e38fe8620d36b4953dfba2c44bb4d50a3048e85d40a343f3dcf5672d307d2ae1"}
Sep 06 13:57:50.706429 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:50.706374    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-52.ec2.internal" podStartSLOduration=6.706350895 podStartE2EDuration="6.706350895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:57:48.700789092 +0000 UTC m=+5.726854268" watchObservedRunningTime="2026-09-06 13:57:50.706350895 +0000 UTC m=+7.732416119"
Sep 06 13:57:51.607345 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:51.607311    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:51.607520 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:51.607456    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:52.604622 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:52.604588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:52.605100 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:52.604713    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:52.605100 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:52.604756    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:52.605100 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:52.604890    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:53.109315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:53.109369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:53.109421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.109566    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.109583    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.109596    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.109655    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:58:01.109635018 +0000 UTC m=+18.135700186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.110098    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.110145    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:01.110130736 +0000 UTC m=+18.136195896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.110202    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:53.110268 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.110235    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:01.110223445 +0000 UTC m=+18.136288599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:57:53.611978 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:53.611950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:53.612429 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:53.612149    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:54.604254 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:54.604218    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:54.604460 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:54.604221    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:54.604460 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:54.604365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:54.604584 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:54.604482    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:55.604361 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:55.604327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:55.604903 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:55.604859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:56.604821 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:56.604732    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:56.605232 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:56.604746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:56.605232 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:56.604865    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:56.605232 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:56.604944    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:57.604911 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:57.604876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:57.605360 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:57.605020    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:57:58.604420 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:58.604386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:57:58.604652 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:58.604519    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:57:58.604652 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:58.604537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:57:58.604783 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:58.604663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:57:59.604396 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:57:59.604362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:57:59.604862 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:57:59.604484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:00.604097 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:00.604054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:00.604327 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:00.604056    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:00.604327 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:00.604176    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:00.604327 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:00.604257    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:01.168270 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:01.168229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:01.168466 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:01.168286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:01.168466 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:01.168318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:01.168466 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168426    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:58:01.168466 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168443    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:58:01.168466 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168458    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:58:01.168718 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168469    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:58:01.168718 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168473    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:58:01.168718 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168505    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:17.168485073 +0000 UTC m=+34.194550240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:58:01.168718 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:17.168512786 +0000 UTC m=+34.194577939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:58:01.168718 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.168539    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:58:17.168529419 +0000 UTC m=+34.194594582 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:58:01.604224 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:01.604138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:01.604397 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:01.604271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:02.604031 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:02.603993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:02.604031 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:02.604022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:02.604554 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:02.604135    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:02.604554 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:02.604250    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:03.604651 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:03.604623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:03.605016 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:03.604706    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:04.605083 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.604897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:04.605543 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.604915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:04.605543 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:04.605186    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:04.605543 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:04.605236    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:04.725749 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.725711    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal" event={"ID":"77b704edf06adae0a8b77a9b0792edab","Type":"ContainerStarted","Data":"16f573544a47687a6f2908d80dae521f67c0e968a35dde83d7cc56bd5fc7773e"}
Sep 06 13:58:04.728162 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.728131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l95xv" event={"ID":"bae7c569-ded6-42e5-9ee9-4c499a27b95b","Type":"ContainerStarted","Data":"bc41f71a7cb9d12abca2612fc988df639cc3f4d8ba632946444f2e401dedba8d"}
Sep 06 13:58:04.729911 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.729882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dzd9g" event={"ID":"fe20f3c5-a2b7-488d-af61-b5845188ff63","Type":"ContainerStarted","Data":"1635b66099b249a998d15d5015da4033b2fed4293386e62aed1e948b9b284042"}
Sep 06 13:58:04.731954 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.731660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhjln" event={"ID":"c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a","Type":"ContainerStarted","Data":"7de3b16a6d96af100ccf1cd9fe7c0ad84a32f276aeb1f81af3765a9993a7f4f3"}
Sep 06 13:58:04.733562 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.733511    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdtzv" event={"ID":"ed6ca2cf-4bc8-4cdf-a430-00968c77f40a","Type":"ContainerStarted","Data":"f1b5d8d228494073452b5046fac33887bc8b5b4bd2afec81697971f42ece916f"}
Sep 06 13:58:04.735506 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.735480    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="7a4dc8f823e7c96af5f390ff46bccebaa0234b0bf0b0e48ed0eaf89eb77bf3fa" exitCode=0
Sep 06 13:58:04.735608 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.735560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"7a4dc8f823e7c96af5f390ff46bccebaa0234b0bf0b0e48ed0eaf89eb77bf3fa"}
Sep 06 13:58:04.744997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.744945    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-52.ec2.internal" podStartSLOduration=20.74492521 podStartE2EDuration="20.74492521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.740030223 +0000 UTC m=+21.766095397" watchObservedRunningTime="2026-09-06 13:58:04.74492521 +0000 UTC m=+21.770990386"
Sep 06 13:58:04.745974 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.745952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 13:58:04.746586 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.746556    2576 generic.go:356] "Generic (PLEG): container finished" podID="079a9bd2-3476-4b43-8f7f-25eaa03baabf" containerID="e819c05e44642241b4ed415ab2352b0f150581eb00c3328670c968a35ad10945" exitCode=1
Sep 06 13:58:04.746714 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.746689    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"14bf6d216e1741d998a6a9ae0059d04b4ba6b80feec87d3e5476277bb3998f1e"}
Sep 06 13:58:04.746920 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.746894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"54f3c1163a6b207dfd59b0825ad5033b61aa61eade3418a0409dd0511ea2a8d8"}
Sep 06 13:58:04.747000 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.746959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"eca5ba9ae09866cd2ea43f7c0d382b68f9d57ae6e8c8aad8d617e35c5f8f9919"}
Sep 06 13:58:04.747049 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.747004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"ddb44d3af2b4f01c2e4324213a5e3a0d3cf73261d1107e2831297df90c23397b"}
Sep 06 13:58:04.747079 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.747061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerDied","Data":"e819c05e44642241b4ed415ab2352b0f150581eb00c3328670c968a35ad10945"}
Sep 06 13:58:04.747121 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.747107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"3a8a4005e9d3930c9fa0f09d02e2d96c388dcf79039ff7e437eea28dba87f045"}
Sep 06 13:58:04.748815 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.748791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz" event={"ID":"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7","Type":"ContainerStarted","Data":"d4d591adcee1e01f2dbbb9eceeb02cae6c46b151fa33f598c17f3b6c502681cd"}
Sep 06 13:58:04.750443 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.750417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nsfxb" event={"ID":"6de74b58-56f8-4cb2-9823-7f4034db8d3d","Type":"ContainerStarted","Data":"5b916fad42a65381797242eebb059e984407f3258a66f8d50c22b019f54dcfe8"}
Sep 06 13:58:04.757851 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.757808    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hhjln" podStartSLOduration=3.077984183 podStartE2EDuration="21.757794802s" totalImagesPullingTime="18.679810619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.757034348 +0000 UTC m=+21.783099522" watchObservedRunningTime="2026-09-06 13:58:04.757794802 +0000 UTC m=+21.783859973"
Sep 06 13:58:04.771640 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.771586    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dzd9g" podStartSLOduration=3.615181577 podStartE2EDuration="21.771564824s" totalImagesPullingTime="18.156383247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.771323207 +0000 UTC m=+21.797388381" watchObservedRunningTime="2026-09-06 13:58:04.771564824 +0000 UTC m=+21.797629999"
Sep 06 13:58:04.784333 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.784284    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l95xv" podStartSLOduration=3.625447529 podStartE2EDuration="21.784266014s" totalImagesPullingTime="18.158818485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.783644595 +0000 UTC m=+21.809709769" watchObservedRunningTime="2026-09-06 13:58:04.784266014 +0000 UTC m=+21.810331190"
Sep 06 13:58:04.818525 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.818467    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdtzv" podStartSLOduration=3.611132798 podStartE2EDuration="21.818450398s" totalImagesPullingTime="18.2073176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.818075593 +0000 UTC m=+21.844140765" watchObservedRunningTime="2026-09-06 13:58:04.818450398 +0000 UTC m=+21.844515573"
Sep 06 13:58:04.831218 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:04.831171    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nsfxb" podStartSLOduration=3.882145775 podStartE2EDuration="20.831160013s" totalImagesPullingTime="16.949014238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:04.830950824 +0000 UTC m=+21.857015998" watchObservedRunningTime="2026-09-06 13:58:04.831160013 +0000 UTC m=+21.857225187"
Sep 06 13:58:05.245249 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.245197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:05.322860 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.322826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:05.522705 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.522673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:58:05.523429 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.523410    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:58:05.577727 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.577433    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 13:58:05.604182 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.604152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:05.604376 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:05.604311    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:05.607058 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.606954    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T13:58:05.577878977Z","UUID":"1147f5d2-5fee-4b56-b5ad-a4d55d8d04de","Handler":null,"Name":"","Endpoint":""}
Sep 06 13:58:05.609231 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.609209    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 13:58:05.609352 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.609246    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 13:58:05.753687 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.753647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qbhlm" event={"ID":"7fe351ac-3d3e-4845-986b-6179ba1e60b7","Type":"ContainerStarted","Data":"573e835e0ccf7fb681b7b452ac3add8e2d7574a1fa1b07f32acd361c310e3714"}
Sep 06 13:58:05.755704 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.755671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz" event={"ID":"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7","Type":"ContainerStarted","Data":"5d1297c60ab8c534b7451e0b4ff741620e3d108170df368dd149e148d0cbc695"}
Sep 06 13:58:05.884242 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.884136    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:05.985047 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:05.985004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:06.604126 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:06.603914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:06.604126 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:06.604056    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:06.604424 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:06.604152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:06.604424 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:06.604291    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:06.758349 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:06.758319    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 13:58:07.608170 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.608132    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:07.608356 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:07.608242    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:07.763963 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.763933    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 13:58:07.764371 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.764346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"3fa1174e57f76464a6ca4bf3ff0d07ca765f0f9eeb7f75807c406a2ee8250847"}
Sep 06 13:58:07.766895 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.766833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz" event={"ID":"48cdb294-0609-4b78-bea2-cd3ff0fbbcf7","Type":"ContainerStarted","Data":"46a2085e1931080a5c5afe62418fa6f1e9d4740f0fad9076ab7920386e5b12a8"}
Sep 06 13:58:07.781363 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.781312    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mk8sz" podStartSLOduration=4.055889886 podStartE2EDuration="24.78129777s" totalImagesPullingTime="20.725407884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:07.781116519 +0000 UTC m=+24.807181695" watchObservedRunningTime="2026-09-06 13:58:07.78129777 +0000 UTC m=+24.807362942"
Sep 06 13:58:07.781544 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:07.781520    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qbhlm" podStartSLOduration=6.613349584 podStartE2EDuration="24.781511804s" totalImagesPullingTime="18.16816222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:05.767375392 +0000 UTC m=+22.793440567" watchObservedRunningTime="2026-09-06 13:58:07.781511804 +0000 UTC m=+24.807576982"
Sep 06 13:58:08.120682 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.120644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:58:08.120882 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.120793    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 13:58:08.121526 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.121497    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nsfxb"
Sep 06 13:58:08.171383 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.171348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:08.400230 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.400073    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:08.472382 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.472347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:08.604930 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.604900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:08.605111 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:08.605021    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:08.605111 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:08.605028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:08.605243 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:08.605128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:09.599822 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.599469    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:09.603991 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.603970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:09.604120 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:09.604062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:09.699102 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.699067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:09.773316 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.773284    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="1f6c8f6f07164a4cc969d240bc9060716a73c98a9ab962728fe348b4339a1004" exitCode=0
Sep 06 13:58:09.773516 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.773369    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"1f6c8f6f07164a4cc969d240bc9060716a73c98a9ab962728fe348b4339a1004"}
Sep 06 13:58:09.777083 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.777063    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 13:58:09.777478 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.777457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"cc5cae6ea38823892eee41eb67753bb90a91aebfe796ff1bd7790616b57a3067"}
Sep 06 13:58:09.777716 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.777696    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:09.777819 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.777744    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:09.777931 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.777912    2576 scope.go:117] "RemoveContainer" containerID="e819c05e44642241b4ed415ab2352b0f150581eb00c3328670c968a35ad10945"
Sep 06 13:58:09.794401 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.794364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:09.794669 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:09.794656    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:10.494787 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.494738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:10.595700 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.595663    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:10.604916 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.604883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:10.604916 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.604914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:10.605541 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:10.605000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:10.605541 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:10.605121    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:10.718939 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.718906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:10.776404 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.776368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log" limit=1024
Sep 06 13:58:10.782105 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.782078    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="457cec73e581de0dad01b1516d788d2f7d4582fa54f2bf4cf49d940797dc2158" exitCode=0
Sep 06 13:58:10.782260 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.782163    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"457cec73e581de0dad01b1516d788d2f7d4582fa54f2bf4cf49d940797dc2158"}
Sep 06 13:58:10.787714 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.787695    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 13:58:10.788077 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.788059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" event={"ID":"079a9bd2-3476-4b43-8f7f-25eaa03baabf","Type":"ContainerStarted","Data":"015780d670f4e072403b9546e58879e402e57883b58b50a2b791f0da419ce358"}
Sep 06 13:58:10.788157 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.788143    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 13:58:10.820984 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.820932    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk" podStartSLOduration=9.54680489 podStartE2EDuration="27.820917189s" totalImagesPullingTime="18.274112299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:10.820025584 +0000 UTC m=+27.846090758" watchObservedRunningTime="2026-09-06 13:58:10.820917189 +0000 UTC m=+27.846982363"
Sep 06 13:58:10.965897 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.965857    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldcbb"]
Sep 06 13:58:10.966073 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.965980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:10.966135 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:10.966095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:10.968720 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.968695    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzmtl"]
Sep 06 13:58:10.968874 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.968796    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:10.968969 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:10.968873    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:10.982726 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.982703    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nmr22"]
Sep 06 13:58:10.982885 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:10.982811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:10.982944 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:10.982883    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:11.696559 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:11.696371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:11.792965 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:11.792929    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="f49e07eb79b1b278fd877110e0c528e24d61ae2497e206cddeb8fd4884ec7423" exitCode=0
Sep 06 13:58:11.793153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:11.793019    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"f49e07eb79b1b278fd877110e0c528e24d61ae2497e206cddeb8fd4884ec7423"}
Sep 06 13:58:12.604383 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:12.604337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:12.604577 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:12.604346    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:12.604577 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:12.604454    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:12.604577 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:12.604337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:12.604577 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:12.604554    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:12.604797 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:12.604632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:14.604841 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:14.604809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:14.604841 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:14.604832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:14.605349 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:14.604832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:14.605349 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:14.604948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:14.605349 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:14.605098    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:14.605349 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:14.605187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:16.604896 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.604789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:16.604896 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.604841    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:16.604896 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.604807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:16.605381 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:16.604915    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lzmtl" podUID="24c1a7f1-7aa3-49ed-b973-3d2b7e8856da"
Sep 06 13:58:16.605381 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:16.604976    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nmr22" podUID="0a218de8-5016-4c86-971d-3f14763198c1"
Sep 06 13:58:16.605381 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:16.605062    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldcbb" podUID="f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7"
Sep 06 13:58:16.807072 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.807041    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="13a6fa973f26cb1fe1278e48e2fc9e9a51b22a37ad9309c4c09cb801a75cf221" exitCode=0
Sep 06 13:58:16.807072 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.807078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"13a6fa973f26cb1fe1278e48e2fc9e9a51b22a37ad9309c4c09cb801a75cf221"}
Sep 06 13:58:16.830430 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:16.830404    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l95xv_bae7c569-ded6-42e5-9ee9-4c499a27b95b/dns-node-resolver/0.log"
Sep 06 13:58:17.186148 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.185964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:17.186148 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186136    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186159    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186171    2576 projected.go:196] Error preparing data for projected volume kube-api-access-mnzd9 for pod openshift-network-diagnostics/network-check-target-lzmtl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186222    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9 podName:24c1a7f1-7aa3-49ed-b973-3d2b7e8856da nodeName:}" failed. No retries permitted until 2026-09-06 13:58:49.186208292 +0000 UTC m=+66.212273445 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mnzd9" (UniqueName: "kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9") pod "network-check-target-lzmtl" (UID: "24c1a7f1-7aa3-49ed-b973-3d2b7e8856da") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186229    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.186169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186280    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret podName:0a218de8-5016-4c86-971d-3f14763198c1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:49.186267533 +0000 UTC m=+66.212332694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret") pod "global-pull-secret-syncer-nmr22" (UID: "0a218de8-5016-4c86-971d-3f14763198c1") : object "kube-system"/"original-pull-secret" not registered
Sep 06 13:58:17.186347 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.186298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:17.186578 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186370    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:58:17.186578 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.186399    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs podName:f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:49.186390643 +0000 UTC m=+66.212455797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs") pod "network-metrics-daemon-ldcbb" (UID: "f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 13:58:17.324132 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.324104    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-52.ec2.internal" event="NodeReady"
Sep 06 13:58:17.324290 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.324241    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 13:58:17.354277 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.354242    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:58:17.374434 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.374406    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rbkk8"]
Sep 06 13:58:17.374602 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.374524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.377084 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.377051    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 13:58:17.377479 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.377060    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kglp\""
Sep 06 13:58:17.377479 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.377067    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 13:58:17.377592 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.377536    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 13:58:17.382312 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.382290    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 13:58:17.388273 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.388243    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5psgm"]
Sep 06 13:58:17.388375 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.388363    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.390645 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.390623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dt8p5\""
Sep 06 13:58:17.390645 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.390642    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 13:58:17.390834 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.390687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 13:58:17.390834 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.390623    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 13:58:17.404336 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.404314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:58:17.404450 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.404348    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbkk8"]
Sep 06 13:58:17.404450 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.404421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.404553 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.404464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5psgm"]
Sep 06 13:58:17.404553 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.404509    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kv2lw"]
Sep 06 13:58:17.406728 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.406683    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 13:58:17.406887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.406774    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 13:58:17.406887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.406797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 13:58:17.406887 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.406776    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-58qh8\""
Sep 06 13:58:17.407518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.407501    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 13:58:17.426495 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.426473    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kv2lw"]
Sep 06 13:58:17.426599 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.426575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.429331 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.429304    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mvgg4\""
Sep 06 13:58:17.429444 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.429366    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 13:58:17.429444 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.429430    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 13:58:17.488691 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.488691 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.488691 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45826550-7beb-46ac-9934-421749d069e2-config-volume\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnhn\" (UniqueName: \"kubernetes.io/projected/45826550-7beb-46ac-9934-421749d069e2-kube-api-access-7gnhn\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbcd790-7eba-4fa1-8e7d-0904c59de236-data-volume\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45826550-7beb-46ac-9934-421749d069e2-tmp-dir\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bcqx\" (UniqueName: \"kubernetes.io/projected/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-api-access-7bcqx\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.488942 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-cert\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.488998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zwb\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjtj\" (UniqueName: \"kubernetes.io/projected/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-kube-api-access-rmjtj\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.489190 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.489409 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.489206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfbcd790-7eba-4fa1-8e7d-0904c59de236-crio-socket\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.590514 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.590514 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bcqx\" (UniqueName: \"kubernetes.io/projected/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-api-access-7bcqx\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-cert\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zwb\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590686    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590755    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls podName:45826550-7beb-46ac-9934-421749d069e2 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:18.090732399 +0000 UTC m=+35.116797566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls") pod "dns-default-kv2lw" (UID: "45826550-7beb-46ac-9934-421749d069e2") : secret "dns-default-metrics-tls" not found
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjtj\" (UniqueName: \"kubernetes.io/projected/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-kube-api-access-rmjtj\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.590803 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590810    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590865    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:18.090849416 +0000 UTC m=+35.116914586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : configmap "kube-rbac-proxy" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590901    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590944    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:18.090930792 +0000 UTC m=+35.116995961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfbcd790-7eba-4fa1-8e7d-0904c59de236-crio-socket\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.590992    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.590996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.591004    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcbd49d64-r8fzm: secret "image-registry-tls" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:17.591071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls podName:10f3c965-7813-4f32-b10d-8e0e4096d9a1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:18.091059716 +0000 UTC m=+35.117124871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls") pod "image-registry-bcbd49d64-r8fzm" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1") : secret "image-registry-tls" not found
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45826550-7beb-46ac-9934-421749d069e2-config-volume\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnhn\" (UniqueName: \"kubernetes.io/projected/45826550-7beb-46ac-9934-421749d069e2-kube-api-access-7gnhn\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbcd790-7eba-4fa1-8e7d-0904c59de236-data-volume\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.591445 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591247    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfbcd790-7eba-4fa1-8e7d-0904c59de236-crio-socket\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45826550-7beb-46ac-9934-421749d069e2-tmp-dir\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfbcd790-7eba-4fa1-8e7d-0904c59de236-data-volume\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45826550-7beb-46ac-9934-421749d069e2-config-volume\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.592131 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.591922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45826550-7beb-46ac-9934-421749d069e2-tmp-dir\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.594983 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.594961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.595095 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.595044    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-cert\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.595095 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.595040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.601732 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.601706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bcqx\" (UniqueName: \"kubernetes.io/projected/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-api-access-7bcqx\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:17.602077 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.602058    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.602285 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.602264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zwb\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:17.602329 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.602264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjtj\" (UniqueName: \"kubernetes.io/projected/d3fe2a27-6404-4a6f-8f57-65a7ac100ee9-kube-api-access-rmjtj\") pod \"ingress-canary-rbkk8\" (UID: \"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9\") " pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.602931 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.602915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnhn\" (UniqueName: \"kubernetes.io/projected/45826550-7beb-46ac-9934-421749d069e2-kube-api-access-7gnhn\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:17.632290 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.632262    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzd9g_fe20f3c5-a2b7-488d-af61-b5845188ff63/node-ca/0.log"
Sep 06 13:58:17.696825 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.696783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbkk8"
Sep 06 13:58:17.813779 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.813737    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f0bd173-2c38-4019-9a81-84cb8910cd0c" containerID="6c0b876b787ea679b798511ea6b8778eabe56c29bc7a4578e440e165a231f581" exitCode=0
Sep 06 13:58:17.813902 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:17.813803    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerDied","Data":"6c0b876b787ea679b798511ea6b8778eabe56c29bc7a4578e440e165a231f581"}
Sep 06 13:58:18.013607 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.013575    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbkk8"]
Sep 06 13:58:18.016810 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:18.016783    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fe2a27_6404_4a6f_8f57_65a7ac100ee9.slice/crio-7ef30d2344fb02ad7972d89f5aab50faba654214c8b596e304496f263523cd11 WatchSource:0}: Error finding container 7ef30d2344fb02ad7972d89f5aab50faba654214c8b596e304496f263523cd11: Status 404 returned error can't find the container with id 7ef30d2344fb02ad7972d89f5aab50faba654214c8b596e304496f263523cd11
Sep 06 13:58:18.094958 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.094862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:18.094958 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.094935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.094975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.094999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095045    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095070    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcbd49d64-r8fzm: secret "image-registry-tls" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095098    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095111    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls podName:10f3c965-7813-4f32-b10d-8e0e4096d9a1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.095117327 +0000 UTC m=+36.121182480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls") pod "image-registry-bcbd49d64-r8fzm" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1") : secret "image-registry-tls" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095101    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 13:58:18.095155 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095154    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.095139512 +0000 UTC m=+36.121204666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:18.095428 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095171    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls podName:45826550-7beb-46ac-9934-421749d069e2 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.09516067 +0000 UTC m=+36.121225829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls") pod "dns-default-kv2lw" (UID: "45826550-7beb-46ac-9934-421749d069e2") : secret "dns-default-metrics-tls" not found
Sep 06 13:58:18.095428 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:18.095187    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:19.095179444 +0000 UTC m=+36.121244598 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : configmap "kube-rbac-proxy" not found
Sep 06 13:58:18.212428 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.212389    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-k8x55"]
Sep 06 13:58:18.232868 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.232840    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-k8x55"]
Sep 06 13:58:18.233022 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.232939    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.235462 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.235287    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 06 13:58:18.235604 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.235352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 06 13:58:18.235604 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.235509    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 06 13:58:18.235801 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.235786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 06 13:58:18.235883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.235865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6v5k8\""
Sep 06 13:58:18.296286 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.296255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-cabundle\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.296490 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.296324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-key\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.296490 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.296343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpcf\" (UniqueName: \"kubernetes.io/projected/0e6f2b12-9934-4f61-ba10-98e014074f97-kube-api-access-9vpcf\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.397372 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.397287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-key\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.397372 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.397326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpcf\" (UniqueName: \"kubernetes.io/projected/0e6f2b12-9934-4f61-ba10-98e014074f97-kube-api-access-9vpcf\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.397588 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.397423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-cabundle\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.398089 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.398066    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-cabundle\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.400077 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.400056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0e6f2b12-9934-4f61-ba10-98e014074f97-signing-key\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.404541 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.404523    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpcf\" (UniqueName: \"kubernetes.io/projected/0e6f2b12-9934-4f61-ba10-98e014074f97-kube-api-access-9vpcf\") pod \"service-ca-75d99c484-k8x55\" (UID: \"0e6f2b12-9934-4f61-ba10-98e014074f97\") " pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.543912 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.543877    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-k8x55"
Sep 06 13:58:18.605807 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.605006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:18.605807 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.605256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:18.605807 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.605725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608003    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608126    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608151    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608260    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\""
Sep 06 13:58:18.608342 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.608283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\""
Sep 06 13:58:18.821487 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.821431    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" event={"ID":"2f0bd173-2c38-4019-9a81-84cb8910cd0c","Type":"ContainerStarted","Data":"5555b3cee92f54a58d199570f899677935185e55c37a9af3ee353bd5fae3a4a5"}
Sep 06 13:58:18.823159 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.823128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbkk8" event={"ID":"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9","Type":"ContainerStarted","Data":"7ef30d2344fb02ad7972d89f5aab50faba654214c8b596e304496f263523cd11"}
Sep 06 13:58:18.839936 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.839896    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-k8x55"]
Sep 06 13:58:18.843376 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:18.843325    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f4xxx" podStartSLOduration=7.9764744400000005 podStartE2EDuration="35.843304797s" totalImagesPullingTime="27.866830357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:18.841905778 +0000 UTC m=+35.867970965" watchObservedRunningTime="2026-09-06 13:58:18.843304797 +0000 UTC m=+35.869369974"
Sep 06 13:58:18.843823 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:18.843797    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6f2b12_9934_4f61_ba10_98e014074f97.slice/crio-eaa926fcbfabb60edb278e810eed2da19423c2de3e9eb8537a1627ec9e726002 WatchSource:0}: Error finding container eaa926fcbfabb60edb278e810eed2da19423c2de3e9eb8537a1627ec9e726002: Status 404 returned error can't find the container with id eaa926fcbfabb60edb278e810eed2da19423c2de3e9eb8537a1627ec9e726002
Sep 06 13:58:19.104988 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:19.104902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:19.104988 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:19.104941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:19.104988 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:19.104974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:19.105040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105044    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105071    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.105110599 +0000 UTC m=+38.131175755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : configmap "kube-rbac-proxy" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105147    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105155    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105175    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcbd49d64-r8fzm: secret "image-registry-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105151    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.105143076 +0000 UTC m=+38.131208231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105208    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls podName:45826550-7beb-46ac-9934-421749d069e2 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.105189988 +0000 UTC m=+38.131255141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls") pod "dns-default-kv2lw" (UID: "45826550-7beb-46ac-9934-421749d069e2") : secret "dns-default-metrics-tls" not found
Sep 06 13:58:19.105404 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:19.105224    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls podName:10f3c965-7813-4f32-b10d-8e0e4096d9a1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:21.105215217 +0000 UTC m=+38.131280374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls") pod "image-registry-bcbd49d64-r8fzm" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1") : secret "image-registry-tls" not found
Sep 06 13:58:19.828102 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:19.828060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-k8x55" event={"ID":"0e6f2b12-9934-4f61-ba10-98e014074f97","Type":"ContainerStarted","Data":"eaa926fcbfabb60edb278e810eed2da19423c2de3e9eb8537a1627ec9e726002"}
Sep 06 13:58:21.122772 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.122725    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.122782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.122867    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.122884    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.122950    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:25.12293038 +0000 UTC m=+42.148995544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.122973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:25.122960298 +0000 UTC m=+42.149025456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : configmap "kube-rbac-proxy" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.122997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.123082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.123166    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.123175    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.123182    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcbd49d64-r8fzm: secret "image-registry-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.123217    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls podName:45826550-7beb-46ac-9934-421749d069e2 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:25.123206078 +0000 UTC m=+42.149271232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls") pod "dns-default-kv2lw" (UID: "45826550-7beb-46ac-9934-421749d069e2") : secret "dns-default-metrics-tls" not found
Sep 06 13:58:21.123246 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:21.123233    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls podName:10f3c965-7813-4f32-b10d-8e0e4096d9a1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:25.123224363 +0000 UTC m=+42.149289516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls") pod "image-registry-bcbd49d64-r8fzm" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1") : secret "image-registry-tls" not found
Sep 06 13:58:21.835366 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.835116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbkk8" event={"ID":"d3fe2a27-6404-4a6f-8f57-65a7ac100ee9","Type":"ContainerStarted","Data":"8e1567f71266607d9b599bcdafb52cb3afaabcffa5dedb8dcf4f1cccec952b36"}
Sep 06 13:58:21.838860 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.838829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-k8x55" event={"ID":"0e6f2b12-9934-4f61-ba10-98e014074f97","Type":"ContainerStarted","Data":"f41cabe45672d7d3be1019e0d2a8a914296c5fef40b892207b3f4c9a393a13e1"}
Sep 06 13:58:21.853086 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.853041    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rbkk8" podStartSLOduration=1.543175544 podStartE2EDuration="4.853025843s" totalImagesPullingTime="3.309850299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:21.852582353 +0000 UTC m=+38.878647528" watchObservedRunningTime="2026-09-06 13:58:21.853025843 +0000 UTC m=+38.879091018"
Sep 06 13:58:21.867590 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:21.867540    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-k8x55" podStartSLOduration=1.035862471 podStartE2EDuration="3.867527745s" totalImagesPullingTime="2.831665274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:21.86704338 +0000 UTC m=+38.893108539" watchObservedRunningTime="2026-09-06 13:58:21.867527745 +0000 UTC m=+38.893592919"
Sep 06 13:58:25.161683 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:25.161639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:25.161683 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:25.161685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.161826    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:25.161832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:25.161895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.161933    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls podName:bfbcd790-7eba-4fa1-8e7d-0904c59de236 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:33.161913972 +0000 UTC m=+50.187979131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5psgm" (UID: "bfbcd790-7eba-4fa1-8e7d-0904c59de236") : secret "insights-runtime-extractor-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.161968    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.161987    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcbd49d64-r8fzm: secret "image-registry-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.161970    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.162046    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls podName:10f3c965-7813-4f32-b10d-8e0e4096d9a1 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:33.162018383 +0000 UTC m=+50.188083541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls") pod "image-registry-bcbd49d64-r8fzm" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1") : secret "image-registry-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:25.162092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls podName:45826550-7beb-46ac-9934-421749d069e2 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:33.162078851 +0000 UTC m=+50.188144005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls") pod "dns-default-kv2lw" (UID: "45826550-7beb-46ac-9934-421749d069e2") : secret "dns-default-metrics-tls" not found
Sep 06 13:58:25.162378 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:25.162348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfbcd790-7eba-4fa1-8e7d-0904c59de236-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:33.225616 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.225578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:33.226395 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.225641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:33.226395 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.225853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:33.229412 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.229371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45826550-7beb-46ac-9934-421749d069e2-metrics-tls\") pod \"dns-default-kv2lw\" (UID: \"45826550-7beb-46ac-9934-421749d069e2\") " pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:33.229412 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.229399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfbcd790-7eba-4fa1-8e7d-0904c59de236-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5psgm\" (UID: \"bfbcd790-7eba-4fa1-8e7d-0904c59de236\") " pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:33.229578 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.229420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"image-registry-bcbd49d64-r8fzm\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") " pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:33.289385 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.289341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:33.312914 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.312881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5psgm"
Sep 06 13:58:33.335583 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.335556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:33.614007 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.613716    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:58:33.617368 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:33.617328    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f3c965_7813_4f32_b10d_8e0e4096d9a1.slice/crio-6b59353914afaabd363ec8e9fa8b0b136d594692d461345a04d8191dd864a1e4 WatchSource:0}: Error finding container 6b59353914afaabd363ec8e9fa8b0b136d594692d461345a04d8191dd864a1e4: Status 404 returned error can't find the container with id 6b59353914afaabd363ec8e9fa8b0b136d594692d461345a04d8191dd864a1e4
Sep 06 13:58:33.641828 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.641797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5psgm"]
Sep 06 13:58:33.649952 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:33.649926    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbcd790_7eba_4fa1_8e7d_0904c59de236.slice/crio-7ec3a3aec1e3884b3bec4cf59020fb7a1775b62e2ef0b48e71be4180fca8e150 WatchSource:0}: Error finding container 7ec3a3aec1e3884b3bec4cf59020fb7a1775b62e2ef0b48e71be4180fca8e150: Status 404 returned error can't find the container with id 7ec3a3aec1e3884b3bec4cf59020fb7a1775b62e2ef0b48e71be4180fca8e150
Sep 06 13:58:33.667977 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.667953    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kv2lw"]
Sep 06 13:58:33.672429 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:33.672406    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45826550_7beb_46ac_9934_421749d069e2.slice/crio-96004e59304cae2699e4e0d8e93aaa33f6f41429b4223639d838fafe7d1b4d71 WatchSource:0}: Error finding container 96004e59304cae2699e4e0d8e93aaa33f6f41429b4223639d838fafe7d1b4d71: Status 404 returned error can't find the container with id 96004e59304cae2699e4e0d8e93aaa33f6f41429b4223639d838fafe7d1b4d71
Sep 06 13:58:33.870452 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.870364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5psgm" event={"ID":"bfbcd790-7eba-4fa1-8e7d-0904c59de236","Type":"ContainerStarted","Data":"8fd3135bcc6673e458eef52054081784cfb2bc443addf2ccfe62d45d1b74ae9a"}
Sep 06 13:58:33.870452 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.870425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5psgm" event={"ID":"bfbcd790-7eba-4fa1-8e7d-0904c59de236","Type":"ContainerStarted","Data":"7ec3a3aec1e3884b3bec4cf59020fb7a1775b62e2ef0b48e71be4180fca8e150"}
Sep 06 13:58:33.871641 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.871605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv2lw" event={"ID":"45826550-7beb-46ac-9934-421749d069e2","Type":"ContainerStarted","Data":"96004e59304cae2699e4e0d8e93aaa33f6f41429b4223639d838fafe7d1b4d71"}
Sep 06 13:58:33.873271 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.873245    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" event={"ID":"10f3c965-7813-4f32-b10d-8e0e4096d9a1","Type":"ContainerStarted","Data":"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"}
Sep 06 13:58:33.873458 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.873287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" event={"ID":"10f3c965-7813-4f32-b10d-8e0e4096d9a1","Type":"ContainerStarted","Data":"6b59353914afaabd363ec8e9fa8b0b136d594692d461345a04d8191dd864a1e4"}
Sep 06 13:58:33.873458 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.873387    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:58:33.889747 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:33.889700    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" podStartSLOduration=37.889683965 podStartE2EDuration="37.889683965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:33.889153132 +0000 UTC m=+50.915218308" watchObservedRunningTime="2026-09-06 13:58:33.889683965 +0000 UTC m=+50.915749140"
Sep 06 13:58:34.880354 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:34.880078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5psgm" event={"ID":"bfbcd790-7eba-4fa1-8e7d-0904c59de236","Type":"ContainerStarted","Data":"9e0bb77cfb46708903a71710bff9119f323e9a555aba77bf1ef7909ba7c4c4b1"}
Sep 06 13:58:36.889044 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.889008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5psgm" event={"ID":"bfbcd790-7eba-4fa1-8e7d-0904c59de236","Type":"ContainerStarted","Data":"37600aa1cf49149b92b891078184cab0f0c6ead9827bfb5b962a9777d9ee15fd"}
Sep 06 13:58:36.890649 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.890619    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv2lw" event={"ID":"45826550-7beb-46ac-9934-421749d069e2","Type":"ContainerStarted","Data":"e046a6d0f491d8fc99eaaa3a2b4e222b5a58224e73a34d6bf5201d61438a52be"}
Sep 06 13:58:36.890792 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.890663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kv2lw" event={"ID":"45826550-7beb-46ac-9934-421749d069e2","Type":"ContainerStarted","Data":"6bd0eb91f9b11ee7c761274f0bdb04c13914b2e4a9104fc5aaf153f358d1ffc2"}
Sep 06 13:58:36.890792 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.890778    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:36.904277 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.904233    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5psgm" podStartSLOduration=17.844252713 podStartE2EDuration="19.904219682s" totalImagesPullingTime="2.059966969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:36.903317518 +0000 UTC m=+53.929382695" watchObservedRunningTime="2026-09-06 13:58:36.904219682 +0000 UTC m=+53.930284850"
Sep 06 13:58:36.918219 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:36.918174    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kv2lw" podStartSLOduration=17.764568312 podStartE2EDuration="19.918158281s" totalImagesPullingTime="2.153589969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:36.917324224 +0000 UTC m=+53.943389400" watchObservedRunningTime="2026-09-06 13:58:36.918158281 +0000 UTC m=+53.944223477"
Sep 06 13:58:39.137689 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.137653    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:58:39.162520 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.162480    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"]
Sep 06 13:58:39.198209 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.198168    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"]
Sep 06 13:58:39.198404 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.198300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.200714 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.200684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 06 13:58:39.200862 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.200724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2lr9j\""
Sep 06 13:58:39.200927 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.200879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 06 13:58:39.201133 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.201119    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 06 13:58:39.201181 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.201149    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 06 13:58:39.215129 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.215103    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"]
Sep 06 13:58:39.215255 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.215223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.217317 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.217295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 06 13:58:39.231824 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.231800    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"]
Sep 06 13:58:39.231997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.231905    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"]
Sep 06 13:58:39.231997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.231933    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"]
Sep 06 13:58:39.231997 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.231905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.236731 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.236708    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 06 13:58:39.236847 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.236717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 06 13:58:39.237029 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.237014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 06 13:58:39.237262 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.237161    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 06 13:58:39.249376 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.249345    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97f89fc94-tvg8r"]
Sep 06 13:58:39.273726 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.273913 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxvl\" (UniqueName: \"kubernetes.io/projected/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-kube-api-access-vtxvl\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.273913 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8f7d59-105a-49cd-b669-ad2300b8a70e-tmp\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.273913 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c8f7d59-105a-49cd-b669-ad2300b8a70e-klusterlet-config\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.273913 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49f4eb23-9024-4adf-bbfc-294fed972bf8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.274062 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.274062 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.273965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.274062 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.274005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrft\" (UniqueName: \"kubernetes.io/projected/49f4eb23-9024-4adf-bbfc-294fed972bf8-kube-api-access-mzrft\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.274062 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.274033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.274062 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.274055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.274232 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.274102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh96k\" (UniqueName: \"kubernetes.io/projected/6c8f7d59-105a-49cd-b669-ad2300b8a70e-kube-api-access-zh96k\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.281724 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.281702    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97f89fc94-tvg8r"]
Sep 06 13:58:39.281832 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.281823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.375171 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-tls\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.375171 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxvl\" (UniqueName: \"kubernetes.io/projected/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-kube-api-access-vtxvl\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8f7d59-105a-49cd-b669-ad2300b8a70e-tmp\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c8f7d59-105a-49cd-b669-ad2300b8a70e-klusterlet-config\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49f4eb23-9024-4adf-bbfc-294fed972bf8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-image-registry-private-configuration\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.375405 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50e352b8-442e-45f2-96ac-5ab8567066b7-ca-trust-extracted\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.375792 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6c8f7d59-105a-49cd-b669-ad2300b8a70e-tmp\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.376000 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.375977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrft\" (UniqueName: \"kubernetes.io/projected/49f4eb23-9024-4adf-bbfc-294fed972bf8-kube-api-access-mzrft\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.376081 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.376125 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.376247 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.376310 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-certificates\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.376598 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-bound-sa-token\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.376598 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-installation-pull-secrets\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.376715 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtz4\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-kube-api-access-lhtz4\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.376715 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-trusted-ca\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.376715 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.376684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh96k\" (UniqueName: \"kubernetes.io/projected/6c8f7d59-105a-49cd-b669-ad2300b8a70e-kube-api-access-zh96k\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.379708 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-ca\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.379883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.379883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.379883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-hub\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.379883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/49f4eb23-9024-4adf-bbfc-294fed972bf8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.379883 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.379840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/6c8f7d59-105a-49cd-b669-ad2300b8a70e-klusterlet-config\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.384984 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.384951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxvl\" (UniqueName: \"kubernetes.io/projected/1a693d93-233d-4b34-8bde-d2e9c1c3ccb2-kube-api-access-vtxvl\") pod \"cluster-proxy-proxy-agent-754d7b764d-8k5hz\" (UID: \"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.385377 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.385360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh96k\" (UniqueName: \"kubernetes.io/projected/6c8f7d59-105a-49cd-b669-ad2300b8a70e-kube-api-access-zh96k\") pod \"klusterlet-addon-workmgr-869689dd46-sg7p8\" (UID: \"6c8f7d59-105a-49cd-b669-ad2300b8a70e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.385551 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.385536    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrft\" (UniqueName: \"kubernetes.io/projected/49f4eb23-9024-4adf-bbfc-294fed972bf8-kube-api-access-mzrft\") pod \"managed-serviceaccount-addon-agent-dd58b5476-gkmw6\" (UID: \"49f4eb23-9024-4adf-bbfc-294fed972bf8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.477867 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-image-registry-private-configuration\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.477867 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50e352b8-442e-45f2-96ac-5ab8567066b7-ca-trust-extracted\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478110 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-certificates\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478110 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-bound-sa-token\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478110 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-installation-pull-secrets\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478110 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.477984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtz4\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-kube-api-access-lhtz4\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478110 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.478059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-trusted-ca\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478345 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.478134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-tls\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478345 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.478292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50e352b8-442e-45f2-96ac-5ab8567066b7-ca-trust-extracted\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.478886 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.478855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-certificates\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.479367 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.479343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50e352b8-442e-45f2-96ac-5ab8567066b7-trusted-ca\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.480384 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.480363    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-installation-pull-secrets\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.480455 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.480394    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50e352b8-442e-45f2-96ac-5ab8567066b7-image-registry-private-configuration\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.480518 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.480503    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-registry-tls\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.484939 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.484920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-bound-sa-token\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.485098 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.485078    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtz4\" (UniqueName: \"kubernetes.io/projected/50e352b8-442e-45f2-96ac-5ab8567066b7-kube-api-access-lhtz4\") pod \"image-registry-97f89fc94-tvg8r\" (UID: \"50e352b8-442e-45f2-96ac-5ab8567066b7\") " pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.522414 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.522384    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"
Sep 06 13:58:39.529540 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.529515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:39.539681 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.539656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"
Sep 06 13:58:39.609508 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.609482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:39.821494 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.821415    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6"]
Sep 06 13:58:39.824359 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:39.824333    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f4eb23_9024_4adf_bbfc_294fed972bf8.slice/crio-87c06422dcf3e8f1bf45f508a7ddb145ffdf1c2347822988df93868cc1c20b1c WatchSource:0}: Error finding container 87c06422dcf3e8f1bf45f508a7ddb145ffdf1c2347822988df93868cc1c20b1c: Status 404 returned error can't find the container with id 87c06422dcf3e8f1bf45f508a7ddb145ffdf1c2347822988df93868cc1c20b1c
Sep 06 13:58:39.847358 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.847332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"]
Sep 06 13:58:39.850337 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:39.850312    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8f7d59_105a_49cd_b669_ad2300b8a70e.slice/crio-714ca33c80a60785f65a1cb4c63642b3e0f318c78bcc8527ed27686783141ae9 WatchSource:0}: Error finding container 714ca33c80a60785f65a1cb4c63642b3e0f318c78bcc8527ed27686783141ae9: Status 404 returned error can't find the container with id 714ca33c80a60785f65a1cb4c63642b3e0f318c78bcc8527ed27686783141ae9
Sep 06 13:58:39.878498 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.878468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz"]
Sep 06 13:58:39.881333 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:39.881309    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a693d93_233d_4b34_8bde_d2e9c1c3ccb2.slice/crio-baf222f9814bd0d1faaa39372b397b2577d4f3bcf38ac1f7310db5c3c08c87ab WatchSource:0}: Error finding container baf222f9814bd0d1faaa39372b397b2577d4f3bcf38ac1f7310db5c3c08c87ab: Status 404 returned error can't find the container with id baf222f9814bd0d1faaa39372b397b2577d4f3bcf38ac1f7310db5c3c08c87ab
Sep 06 13:58:39.901168 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.901140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz" event={"ID":"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2","Type":"ContainerStarted","Data":"baf222f9814bd0d1faaa39372b397b2577d4f3bcf38ac1f7310db5c3c08c87ab"}
Sep 06 13:58:39.902411 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.902388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8" event={"ID":"6c8f7d59-105a-49cd-b669-ad2300b8a70e","Type":"ContainerStarted","Data":"714ca33c80a60785f65a1cb4c63642b3e0f318c78bcc8527ed27686783141ae9"}
Sep 06 13:58:39.903456 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.903437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6" event={"ID":"49f4eb23-9024-4adf-bbfc-294fed972bf8","Type":"ContainerStarted","Data":"87c06422dcf3e8f1bf45f508a7ddb145ffdf1c2347822988df93868cc1c20b1c"}
Sep 06 13:58:39.917789 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:39.917745    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97f89fc94-tvg8r"]
Sep 06 13:58:39.920360 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:39.920337    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e352b8_442e_45f2_96ac_5ab8567066b7.slice/crio-52422b4d74dea047345c15e8e960f82af9447d7cae92f4d026a7cb0400ccc0a8 WatchSource:0}: Error finding container 52422b4d74dea047345c15e8e960f82af9447d7cae92f4d026a7cb0400ccc0a8: Status 404 returned error can't find the container with id 52422b4d74dea047345c15e8e960f82af9447d7cae92f4d026a7cb0400ccc0a8
Sep 06 13:58:40.917306 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:40.916528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r" event={"ID":"50e352b8-442e-45f2-96ac-5ab8567066b7","Type":"ContainerStarted","Data":"28af949b1fdc63f286e07a250b211de546f5bf460afb2d579f4d9af36b72dbdf"}
Sep 06 13:58:40.917306 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:40.916690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r" event={"ID":"50e352b8-442e-45f2-96ac-5ab8567066b7","Type":"ContainerStarted","Data":"52422b4d74dea047345c15e8e960f82af9447d7cae92f4d026a7cb0400ccc0a8"}
Sep 06 13:58:40.917306 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:40.917083    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:58:40.936035 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:40.935922    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r" podStartSLOduration=1.9359021749999998 podStartE2EDuration="1.935902175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:40.93466542 +0000 UTC m=+57.960730596" watchObservedRunningTime="2026-09-06 13:58:40.935902175 +0000 UTC m=+57.961967345"
Sep 06 13:58:42.820235 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:42.819928    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ntgbk"
Sep 06 13:58:46.896313 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:46.896282    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kv2lw"
Sep 06 13:58:47.945839 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.945794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz" event={"ID":"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2","Type":"ContainerStarted","Data":"0724bb60d026031b8f38521ada59020a7c1e6d1f2f3dad7f9483c654af8eccaf"}
Sep 06 13:58:47.945839 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.945845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz" event={"ID":"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2","Type":"ContainerStarted","Data":"144f51d339bd3a4260f1f92fcbb5d917bcad2ad27eadc6288a1530bba673383b"}
Sep 06 13:58:47.946352 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.945865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz" event={"ID":"1a693d93-233d-4b34-8bde-d2e9c1c3ccb2","Type":"ContainerStarted","Data":"ca4625c53b1c7d8fafac1685e765970d960e96d5b1a6e331c20b495095dc9f15"}
Sep 06 13:58:47.947256 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.947224    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8" event={"ID":"6c8f7d59-105a-49cd-b669-ad2300b8a70e","Type":"ContainerStarted","Data":"62f0f82cffa62821aa791cbce590ac1a05ab2ceaa5ac747c6003e29ac2dc4d7f"}
Sep 06 13:58:47.947385 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.947358    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:47.948784 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.948743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6" event={"ID":"49f4eb23-9024-4adf-bbfc-294fed972bf8","Type":"ContainerStarted","Data":"b60125a51a56cf0ba7c9805e13dac815908fc32d65417aa414fe71650b4e51a0"}
Sep 06 13:58:47.949297 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.949279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8"
Sep 06 13:58:47.964246 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.964207    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-754d7b764d-8k5hz" podStartSLOduration=1.7384543460000002 podStartE2EDuration="8.964196093s" totalImagesPullingTime="7.225741747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:47.963143323 +0000 UTC m=+64.989208497" watchObservedRunningTime="2026-09-06 13:58:47.964196093 +0000 UTC m=+64.990261269"
Sep 06 13:58:47.976487 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.976439    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869689dd46-sg7p8" podStartSLOduration=1.706719706 podStartE2EDuration="8.976423506s" totalImagesPullingTime="7.2697038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:47.976185442 +0000 UTC m=+65.002250621" watchObservedRunningTime="2026-09-06 13:58:47.976423506 +0000 UTC m=+65.002488681"
Sep 06 13:58:47.993741 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:47.993697    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd58b5476-gkmw6" podStartSLOduration=1.711439338 podStartE2EDuration="8.993687239s" totalImagesPullingTime="7.282247901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:47.993149349 +0000 UTC m=+65.019214530" watchObservedRunningTime="2026-09-06 13:58:47.993687239 +0000 UTC m=+65.019752414"
Sep 06 13:58:49.144279 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.144239    2576 patch_prober.go:28] interesting pod/image-registry-bcbd49d64-r8fzm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 06 13:58:49.144664 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.144304    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 13:58:49.269900 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.269859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:49.269900 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.269903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:49.270153 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.270020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:49.272097 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.272078    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 13:58:49.272812 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.272792    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 13:58:49.272902 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.272801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 13:58:49.282466 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.282438    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 13:58:49.282924 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.282904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0a218de8-5016-4c86-971d-3f14763198c1-original-pull-secret\") pod \"global-pull-secret-syncer-nmr22\" (UID: \"0a218de8-5016-4c86-971d-3f14763198c1\") " pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:49.282995 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.282938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7-metrics-certs\") pod \"network-metrics-daemon-ldcbb\" (UID: \"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7\") " pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:49.293256 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.293227    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzd9\" (UniqueName: \"kubernetes.io/projected/24c1a7f1-7aa3-49ed-b973-3d2b7e8856da-kube-api-access-mnzd9\") pod \"network-check-target-lzmtl\" (UID: \"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da\") " pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:49.521983 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.521949    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-htlhn\""
Sep 06 13:58:49.528799 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.528754    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nmr22"
Sep 06 13:58:49.530067 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.530049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:49.540321 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.540298    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8cxw7\""
Sep 06 13:58:49.570109 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.563850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldcbb"
Sep 06 13:58:49.820638 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.820565    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nmr22"]
Sep 06 13:58:49.823824 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:49.823799    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a218de8_5016_4c86_971d_3f14763198c1.slice/crio-b943f8ee15c2f6207aa7ae44e3396c54ff6e2672a7e8b67c2394f5406c97e463 WatchSource:0}: Error finding container b943f8ee15c2f6207aa7ae44e3396c54ff6e2672a7e8b67c2394f5406c97e463: Status 404 returned error can't find the container with id b943f8ee15c2f6207aa7ae44e3396c54ff6e2672a7e8b67c2394f5406c97e463
Sep 06 13:58:49.839025 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.838954    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lzmtl"]
Sep 06 13:58:49.841336 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:49.841307    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c1a7f1_7aa3_49ed_b973_3d2b7e8856da.slice/crio-d2d1f3305a8920dfa5e2307d0711fb1d3d57c3a9abd431ccf5c30bde71887c05 WatchSource:0}: Error finding container d2d1f3305a8920dfa5e2307d0711fb1d3d57c3a9abd431ccf5c30bde71887c05: Status 404 returned error can't find the container with id d2d1f3305a8920dfa5e2307d0711fb1d3d57c3a9abd431ccf5c30bde71887c05
Sep 06 13:58:49.853756 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.853732    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldcbb"]
Sep 06 13:58:49.856938 ip-10-0-133-52 kubenswrapper[2576]: W0906 13:58:49.856909    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cc8ab5_9e3c_4e89_81a5_c56822e7bbf7.slice/crio-57c38edfd952675665d0f459833d460fe744e03e33a9490c9a4748e72a62fc5e WatchSource:0}: Error finding container 57c38edfd952675665d0f459833d460fe744e03e33a9490c9a4748e72a62fc5e: Status 404 returned error can't find the container with id 57c38edfd952675665d0f459833d460fe744e03e33a9490c9a4748e72a62fc5e
Sep 06 13:58:49.956922 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.956880    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzmtl" event={"ID":"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da","Type":"ContainerStarted","Data":"d2d1f3305a8920dfa5e2307d0711fb1d3d57c3a9abd431ccf5c30bde71887c05"}
Sep 06 13:58:49.958063 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.958040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldcbb" event={"ID":"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7","Type":"ContainerStarted","Data":"57c38edfd952675665d0f459833d460fe744e03e33a9490c9a4748e72a62fc5e"}
Sep 06 13:58:49.959237 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:49.959216    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nmr22" event={"ID":"0a218de8-5016-4c86-971d-3f14763198c1","Type":"ContainerStarted","Data":"b943f8ee15c2f6207aa7ae44e3396c54ff6e2672a7e8b67c2394f5406c97e463"}
Sep 06 13:58:50.232993 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.232513    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ht946"]
Sep 06 13:58:50.248955 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.248457    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.251090 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.251063    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zxjvq\""
Sep 06 13:58:50.257261 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.257231    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 13:58:50.257785 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.257446    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 13:58:50.257785 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.257646    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 13:58:50.257932 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.257825    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 13:58:50.258008 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.257992    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 13:58:50.258182 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.258168    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-root\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-accelerators-collector-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-textfile\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-wtmp\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-sys\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cv9v\" (UniqueName: \"kubernetes.io/projected/5be1a55b-675c-4f8f-bae5-20aa46ef7298-kube-api-access-4cv9v\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.379515 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.379396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-metrics-client-ca\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480247 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-sys\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cv9v\" (UniqueName: \"kubernetes.io/projected/5be1a55b-675c-4f8f-bae5-20aa46ef7298-kube-api-access-4cv9v\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-metrics-client-ca\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-sys\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480432 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-root\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:50.480454    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-accelerators-collector-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:58:50.480517    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls podName:5be1a55b-675c-4f8f-bae5-20aa46ef7298 nodeName:}" failed. No retries permitted until 2026-09-06 13:58:50.980494581 +0000 UTC m=+68.006559738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls") pod "node-exporter-ht946" (UID: "5be1a55b-675c-4f8f-bae5-20aa46ef7298") : secret "node-exporter-tls" not found
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-root\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-textfile\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.480716 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-wtmp\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.481036 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.480741    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-wtmp\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.481223 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.481198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-accelerators-collector-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.481561 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.481539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-textfile\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.481690 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.481667    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be1a55b-675c-4f8f-bae5-20aa46ef7298-metrics-client-ca\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.484050 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.483895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.488733 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.488681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cv9v\" (UniqueName: \"kubernetes.io/projected/5be1a55b-675c-4f8f-bae5-20aa46ef7298-kube-api-access-4cv9v\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.985838 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.985805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:50.988858 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:50.988830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5be1a55b-675c-4f8f-bae5-20aa46ef7298-node-exporter-tls\") pod \"node-exporter-ht946\" (UID: \"5be1a55b-675c-4f8f-bae5-20aa46ef7298\") " pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:51.165250 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:51.165175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ht946"
Sep 06 13:58:51.971178 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:51.970692    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldcbb" event={"ID":"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7","Type":"ContainerStarted","Data":"8c58f7b9106f8f1b104b1ff1684c40bfeba6316eac1e302d1a09a159ed3133d8"}
Sep 06 13:58:51.971178 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:51.970787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldcbb" event={"ID":"f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7","Type":"ContainerStarted","Data":"838f810850f3ba17cffec850f0c9fb71221eb387bf0d9b328c897cce9bc1c015"}
Sep 06 13:58:51.973811 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:51.973782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ht946" event={"ID":"5be1a55b-675c-4f8f-bae5-20aa46ef7298","Type":"ContainerStarted","Data":"2ab9731ee5cf730c733383808cebcdf732694747eaa0246634e7f4cca2218ff9"}
Sep 06 13:58:51.986427 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:51.986362    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldcbb" podStartSLOduration=67.89134035 podStartE2EDuration="1m8.986316088s" totalImagesPullingTime="1.094975738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:51.985532153 +0000 UTC m=+69.011597330" watchObservedRunningTime="2026-09-06 13:58:51.986316088 +0000 UTC m=+69.012381265"
Sep 06 13:58:52.979965 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:52.979916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ht946" event={"ID":"5be1a55b-675c-4f8f-bae5-20aa46ef7298","Type":"ContainerStarted","Data":"746940e47f5f300ffd733974b6093fcfa9120d812f6e05cbdd12aa6861e1dee8"}
Sep 06 13:58:53.985925 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:53.985885    2576 generic.go:356] "Generic (PLEG): container finished" podID="5be1a55b-675c-4f8f-bae5-20aa46ef7298" containerID="746940e47f5f300ffd733974b6093fcfa9120d812f6e05cbdd12aa6861e1dee8" exitCode=0
Sep 06 13:58:53.986377 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:53.985941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ht946" event={"ID":"5be1a55b-675c-4f8f-bae5-20aa46ef7298","Type":"ContainerDied","Data":"746940e47f5f300ffd733974b6093fcfa9120d812f6e05cbdd12aa6861e1dee8"}
Sep 06 13:58:55.996577 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:55.996533    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nmr22" event={"ID":"0a218de8-5016-4c86-971d-3f14763198c1","Type":"ContainerStarted","Data":"63671b99d0161be8856ea6e970dea38e1d8d1d871892080aa954a27b1ddd0c34"}
Sep 06 13:58:55.998754 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:55.998718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ht946" event={"ID":"5be1a55b-675c-4f8f-bae5-20aa46ef7298","Type":"ContainerStarted","Data":"4966e5a4b58a7532737abba8bcd4a88830d8476807bdc770b54b074e9a6be2d6"}
Sep 06 13:58:55.998911 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:55.998809    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ht946" event={"ID":"5be1a55b-675c-4f8f-bae5-20aa46ef7298","Type":"ContainerStarted","Data":"a42592f7b2a408db1e1c2eca497882536171d90c42043b9b944212cf8bf70bf3"}
Sep 06 13:58:56.000245 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:56.000226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lzmtl" event={"ID":"24c1a7f1-7aa3-49ed-b973-3d2b7e8856da","Type":"ContainerStarted","Data":"d450a887fad8b5434b25d5e77f4946a42d91370938671275d347eee90e864780"}
Sep 06 13:58:56.000390 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:56.000373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 13:58:56.010261 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:56.010206    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nmr22" podStartSLOduration=66.898090507 podStartE2EDuration="1m12.010186831s" totalImagesPullingTime="5.112096324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 13:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:56.009198869 +0000 UTC m=+73.035264045" watchObservedRunningTime="2026-09-06 13:58:56.010186831 +0000 UTC m=+73.036252007"
Sep 06 13:58:56.024747 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:56.024689    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ht946" podStartSLOduration=4.893350374 podStartE2EDuration="6.024675873s" totalImagesPullingTime="1.053642458s" totalInitContainerRuntime="77.683041ms" isStatefulPod=true podCreationTimestamp="2026-09-06 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:56.024292306 +0000 UTC m=+73.050357493" watchObservedRunningTime="2026-09-06 13:58:56.024675873 +0000 UTC m=+73.050741050"
Sep 06 13:58:56.037741 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:56.037689    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lzmtl" podStartSLOduration=67.946803656 podStartE2EDuration="1m13.037672381s" totalImagesPullingTime="5.090868725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 13:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 13:58:56.037225766 +0000 UTC m=+73.063290940" watchObservedRunningTime="2026-09-06 13:58:56.037672381 +0000 UTC m=+73.063737556"
Sep 06 13:58:59.142738 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:58:59.142708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:59:01.927240 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:01.927206    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97f89fc94-tvg8r"
Sep 06 13:59:04.157398 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.157347    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerName="registry" containerID="cri-o://50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159" gracePeriod=30
Sep 06 13:59:04.400376 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.400350    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:59:04.502948 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.502909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.502948 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.502948    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503200 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.502972    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503200 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.502990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503200 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503129    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503200 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503398 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503200    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503398 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503296    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2zwb\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb\") pod \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\" (UID: \"10f3c965-7813-4f32-b10d-8e0e4096d9a1\") "
Sep 06 13:59:04.503554 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503446    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 13:59:04.503619 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.503568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 13:59:04.505562 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.505524    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb" (OuterVolumeSpecName: "kube-api-access-g2zwb") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "kube-api-access-g2zwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 13:59:04.505724 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.505613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 13:59:04.505724 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.505613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 13:59:04.505724 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.505698    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 13:59:04.506058 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.506034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 13:59:04.511831 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.511795    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "10f3c965-7813-4f32-b10d-8e0e4096d9a1" (UID: "10f3c965-7813-4f32-b10d-8e0e4096d9a1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 13:59:04.604325 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604287    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-bound-sa-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604325 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604316    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-trusted-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604325 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604325    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-certificates\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604325 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604334    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-registry-tls\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604578 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604342    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10f3c965-7813-4f32-b10d-8e0e4096d9a1-ca-trust-extracted\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604578 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604350    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2zwb\" (UniqueName: \"kubernetes.io/projected/10f3c965-7813-4f32-b10d-8e0e4096d9a1-kube-api-access-g2zwb\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604578 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604360    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-image-registry-private-configuration\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:04.604578 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:04.604369    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10f3c965-7813-4f32-b10d-8e0e4096d9a1-installation-pull-secrets\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 13:59:05.034641 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.034602    2576 generic.go:356] "Generic (PLEG): container finished" podID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerID="50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159" exitCode=0
Sep 06 13:59:05.034641 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.034644    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm"
Sep 06 13:59:05.034959 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.034685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" event={"ID":"10f3c965-7813-4f32-b10d-8e0e4096d9a1","Type":"ContainerDied","Data":"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"}
Sep 06 13:59:05.034959 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.034731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcbd49d64-r8fzm" event={"ID":"10f3c965-7813-4f32-b10d-8e0e4096d9a1","Type":"ContainerDied","Data":"6b59353914afaabd363ec8e9fa8b0b136d594692d461345a04d8191dd864a1e4"}
Sep 06 13:59:05.035456 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.035442    2576 scope.go:117] "RemoveContainer" containerID="50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"
Sep 06 13:59:05.043938 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.043915    2576 scope.go:117] "RemoveContainer" containerID="50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"
Sep 06 13:59:05.044244 ip-10-0-133-52 kubenswrapper[2576]: E0906 13:59:05.044224    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159\": container with ID starting with 50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159 not found: ID does not exist" containerID="50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"
Sep 06 13:59:05.044316 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.044252    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159"} err="failed to get container status \"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159\": rpc error: code = NotFound desc = could not find container \"50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159\": container with ID starting with 50d1f67fd84c6f23a8d785b0647867d8760efb481c6a33bf5c98e9bebc18e159 not found: ID does not exist"
Sep 06 13:59:05.054310 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.054271    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:59:05.057992 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.057953    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bcbd49d64-r8fzm"]
Sep 06 13:59:05.609319 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:05.609287    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" path="/var/lib/kubelet/pods/10f3c965-7813-4f32-b10d-8e0e4096d9a1/volumes"
Sep 06 13:59:27.007306 ip-10-0-133-52 kubenswrapper[2576]: I0906 13:59:27.007270    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lzmtl"
Sep 06 14:02:43.406628 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:02:43.406601    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:02:43.409725 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:02:43.409709    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 14:02:43.546415 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:02:43.546376    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:03:16.764245 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.764206    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"]
Sep 06 14:03:16.766613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.764455    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerName="registry"
Sep 06 14:03:16.766613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.764467    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerName="registry"
Sep 06 14:03:16.766613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.764518    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="10f3c965-7813-4f32-b10d-8e0e4096d9a1" containerName="registry"
Sep 06 14:03:16.767514 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.767499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.770333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.770304    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 14:03:16.770333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.770304    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 14:03:16.771298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.771264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c7zg9\""
Sep 06 14:03:16.771298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.771276    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 14:03:16.771470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.771265    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 14:03:16.774125 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.774105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.774209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.774137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.774209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.774163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zmn\" (UniqueName: \"kubernetes.io/projected/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-kube-api-access-h8zmn\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.774305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.774240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.774305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.774280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.781890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.781864    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"]
Sep 06 14:03:16.875361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875328    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.875361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.875612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zmn\" (UniqueName: \"kubernetes.io/projected/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-kube-api-access-h8zmn\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.875612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.875612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.875834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.875805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-manifests\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.877777 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.877745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-tmp\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.877896 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.877872    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.878114 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.878092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:16.884350 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:16.884325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zmn\" (UniqueName: \"kubernetes.io/projected/ada67cdc-a7f0-4d66-b335-ac99cdf322d0-kube-api-access-h8zmn\") pod \"opendatahub-operator-controller-manager-7b6858f974-w7ptw\" (UID: \"ada67cdc-a7f0-4d66-b335-ac99cdf322d0\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:17.077832 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:17.077736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:17.351188 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:17.351087    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"]
Sep 06 14:03:17.355994 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:17.355962    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada67cdc_a7f0_4d66_b335_ac99cdf322d0.slice/crio-d0e6029206ccaba16e830330828958f8dfd91cd39e213d595747190890472c8f WatchSource:0}: Error finding container d0e6029206ccaba16e830330828958f8dfd91cd39e213d595747190890472c8f: Status 404 returned error can't find the container with id d0e6029206ccaba16e830330828958f8dfd91cd39e213d595747190890472c8f
Sep 06 14:03:17.357940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:17.357924    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 14:03:17.876938 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:17.876904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw" event={"ID":"ada67cdc-a7f0-4d66-b335-ac99cdf322d0","Type":"ContainerStarted","Data":"d0e6029206ccaba16e830330828958f8dfd91cd39e213d595747190890472c8f"}
Sep 06 14:03:20.890655 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:20.890618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw" event={"ID":"ada67cdc-a7f0-4d66-b335-ac99cdf322d0","Type":"ContainerStarted","Data":"f2d7ba3d9670bc95055de18456119f19b2cb2f67127cbe8f38e2b761701d7685"}
Sep 06 14:03:21.895158 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:21.895123    2576 generic.go:356] "Generic (PLEG): container finished" podID="ada67cdc-a7f0-4d66-b335-ac99cdf322d0" containerID="f2d7ba3d9670bc95055de18456119f19b2cb2f67127cbe8f38e2b761701d7685" exitCode=0
Sep 06 14:03:21.895536 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:21.895204    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw" event={"ID":"ada67cdc-a7f0-4d66-b335-ac99cdf322d0","Type":"ContainerDied","Data":"f2d7ba3d9670bc95055de18456119f19b2cb2f67127cbe8f38e2b761701d7685"}
Sep 06 14:03:22.902410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:22.902372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw" event={"ID":"ada67cdc-a7f0-4d66-b335-ac99cdf322d0","Type":"ContainerStarted","Data":"b4eca87ee996d5536c6510b553cf81f06c1b0100fb8d673a699e1a12ab6486e7"}
Sep 06 14:03:22.902895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:22.902492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:22.920525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:22.920478    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw" podStartSLOduration=2.679401167 podStartE2EDuration="6.920463296s" totalImagesPullingTime="3.236926223s" totalInitContainerRuntime="1.004135906s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:22.920347909 +0000 UTC m=+339.946413087" watchObservedRunningTime="2026-09-06 14:03:22.920463296 +0000 UTC m=+339.946528470"
Sep 06 14:03:33.907092 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:33.907058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6858f974-w7ptw"
Sep 06 14:03:35.798805 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.798696    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"]
Sep 06 14:03:35.807756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.807730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:35.810300 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.810276    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 14:03:35.810606 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.810584    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 14:03:35.811159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.811138    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 14:03:35.811260 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.811167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 14:03:35.811260 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.811167    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8wpml\""
Sep 06 14:03:35.816277 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.816246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"]
Sep 06 14:03:35.911217 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.911177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:35.911413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.911238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tmp\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:35.911413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:35.911298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2fb\" (UniqueName: \"kubernetes.io/projected/b02edfcb-2107-4b0c-a061-91ef848f3dc6-kube-api-access-gd2fb\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.012199 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.012152    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.012369 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:03:36.012308    2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 06 14:03:36.012369 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.012322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tmp\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.012443 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:03:36.012385    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs podName:b02edfcb-2107-4b0c-a061-91ef848f3dc6 nodeName:}" failed. No retries permitted until 2026-09-06 14:03:36.512362887 +0000 UTC m=+353.538428049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs") pod "kube-auth-proxy-86996bf9f5-wz2lc" (UID: "b02edfcb-2107-4b0c-a061-91ef848f3dc6") : secret "kube-auth-proxy-tls" not found
Sep 06 14:03:36.012443 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.012401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2fb\" (UniqueName: \"kubernetes.io/projected/b02edfcb-2107-4b0c-a061-91ef848f3dc6-kube-api-access-gd2fb\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.014543 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.014518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tmp\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.020298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.020273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2fb\" (UniqueName: \"kubernetes.io/projected/b02edfcb-2107-4b0c-a061-91ef848f3dc6-kube-api-access-gd2fb\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.516215 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.516173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.518777 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.518742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02edfcb-2107-4b0c-a061-91ef848f3dc6-tls-certs\") pod \"kube-auth-proxy-86996bf9f5-wz2lc\" (UID: \"b02edfcb-2107-4b0c-a061-91ef848f3dc6\") " pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.718058 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.718022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"
Sep 06 14:03:36.996870 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:36.996834    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc"]
Sep 06 14:03:37.000618 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:37.000591    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02edfcb_2107_4b0c_a061_91ef848f3dc6.slice/crio-0789ac421cf7e4a23f0c88d2a5d8f896dcfe3674d45358c969ef72ef81a6f4f4 WatchSource:0}: Error finding container 0789ac421cf7e4a23f0c88d2a5d8f896dcfe3674d45358c969ef72ef81a6f4f4: Status 404 returned error can't find the container with id 0789ac421cf7e4a23f0c88d2a5d8f896dcfe3674d45358c969ef72ef81a6f4f4
Sep 06 14:03:37.957681 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:37.957637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc" event={"ID":"b02edfcb-2107-4b0c-a061-91ef848f3dc6","Type":"ContainerStarted","Data":"0789ac421cf7e4a23f0c88d2a5d8f896dcfe3674d45358c969ef72ef81a6f4f4"}
Sep 06 14:03:39.967525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:39.967428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc" event={"ID":"b02edfcb-2107-4b0c-a061-91ef848f3dc6","Type":"ContainerStarted","Data":"fb2629a237430601465293be390e1aa526e26c7ad102670766558007a3977b3b"}
Sep 06 14:03:39.982507 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:39.982451    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86996bf9f5-wz2lc" podStartSLOduration=2.288957652 podStartE2EDuration="4.982434855s" totalImagesPullingTime="2.693477203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:39.981569796 +0000 UTC m=+357.007634971" watchObservedRunningTime="2026-09-06 14:03:39.982434855 +0000 UTC m=+357.008500029"
Sep 06 14:03:43.019694 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.019654    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:43.023299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.023270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.025371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.025351    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 14:03:43.025500 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.025445    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-h8jkm\""
Sep 06 14:03:43.035308 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.035275    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:43.063948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.063909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.063948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.063945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064158 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064005    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064158 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064158 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4fw\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064158 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.064310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.064290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.111654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.111622    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"]
Sep 06 14:03:43.114625 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.114608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.126815 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.126790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"]
Sep 06 14:03:43.165220 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad195460-b4fc-49e1-af85-2b9a9e228fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.165411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.165411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4fw\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctz5\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-kube-api-access-zctz5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.165622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.165799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.166440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.166091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.166440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.166251    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.167951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.167923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.168160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.168142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.172175 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.172146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.172417 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.172398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4fw\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bctkc8q\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.267179 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctz5\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-kube-api-access-zctz5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267179 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad195460-b4fc-49e1-af85-2b9a9e228fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267495    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.267825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.267797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.268082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.268060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.268166 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.268082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.268221 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.268167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad195460-b4fc-49e1-af85-2b9a9e228fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.269648 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.269617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.269807 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.269745    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.275443 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.275416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.275688 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.275666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctz5\" (UniqueName: \"kubernetes.io/projected/ad195460-b4fc-49e1-af85-2b9a9e228fca-kube-api-access-zctz5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw\" (UID: \"ad195460-b4fc-49e1-af85-2b9a9e228fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.333168 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.333131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:43.424268 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.424237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:43.609869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.609797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:43.612712 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:43.612682    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab4a48b_5b15_4415_9f85_d568e51e8dfb.slice/crio-79cd6f3925454dbbeb5830720e393e2d07a1caa086a977107c5402e2b1a481a0 WatchSource:0}: Error finding container 79cd6f3925454dbbeb5830720e393e2d07a1caa086a977107c5402e2b1a481a0: Status 404 returned error can't find the container with id 79cd6f3925454dbbeb5830720e393e2d07a1caa086a977107c5402e2b1a481a0
Sep 06 14:03:43.695069 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.695039    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"]
Sep 06 14:03:43.697441 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:43.697412    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad195460_b4fc_49e1_af85_2b9a9e228fca.slice/crio-dcbdd6f35380ce728247e078a480137dd4b64227295181901a7a7a22784b338d WatchSource:0}: Error finding container dcbdd6f35380ce728247e078a480137dd4b64227295181901a7a7a22784b338d: Status 404 returned error can't find the container with id dcbdd6f35380ce728247e078a480137dd4b64227295181901a7a7a22784b338d
Sep 06 14:03:43.982979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.982934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" event={"ID":"fab4a48b-5b15-4415-9f85-d568e51e8dfb","Type":"ContainerStarted","Data":"79cd6f3925454dbbeb5830720e393e2d07a1caa086a977107c5402e2b1a481a0"}
Sep 06 14:03:43.984030 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:43.984007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw" event={"ID":"ad195460-b4fc-49e1-af85-2b9a9e228fca","Type":"ContainerStarted","Data":"dcbdd6f35380ce728247e078a480137dd4b64227295181901a7a7a22784b338d"}
Sep 06 14:03:46.482301 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.482236    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.482750 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.482347    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.482750 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.482392    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.495299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.495260    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.495412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.495349    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.495412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.495392    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:03:46.997035 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:46.997000    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9285d"]
Sep 06 14:03:47.000428 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.000399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw" event={"ID":"ad195460-b4fc-49e1-af85-2b9a9e228fca","Type":"ContainerStarted","Data":"bb0b9361f1e21bbd61b7c42369c1f32a7a3f3be5d02b77946447b0a7cb67d791"}
Sep 06 14:03:47.000583 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.000518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.000829 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.000803    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" event={"ID":"fab4a48b-5b15-4415-9f85-d568e51e8dfb","Type":"ContainerStarted","Data":"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"}
Sep 06 14:03:47.002643 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.002620    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rzt8g\""
Sep 06 14:03:47.002754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.002654    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 06 14:03:47.009895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.009871    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9285d"]
Sep 06 14:03:47.023387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.023332    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw" podStartSLOduration=1.240686831 podStartE2EDuration="4.023316138s" totalImagesPullingTime="2.782629307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:47.022418979 +0000 UTC m=+364.048484157" watchObservedRunningTime="2026-09-06 14:03:47.023316138 +0000 UTC m=+364.049381312"
Sep 06 14:03:47.043305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.043257    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" podStartSLOduration=1.163116326 podStartE2EDuration="4.04324138s" totalImagesPullingTime="2.880125054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:47.041875116 +0000 UTC m=+364.067940291" watchObservedRunningTime="2026-09-06 14:03:47.04324138 +0000 UTC m=+364.069306554"
Sep 06 14:03:47.099616 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.099576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.100033 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.100009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7w5\" (UniqueName: \"kubernetes.io/projected/9e8488b3-e109-41ae-acb2-077daf435dfc-kube-api-access-lr7w5\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.200951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.200914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.200951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.200955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7w5\" (UniqueName: \"kubernetes.io/projected/9e8488b3-e109-41ae-acb2-077daf435dfc-kube-api-access-lr7w5\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.201188 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:03:47.201057    2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 06 14:03:47.201188 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:03:47.201117    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs podName:9e8488b3-e109-41ae-acb2-077daf435dfc nodeName:}" failed. No retries permitted until 2026-09-06 14:03:47.701101335 +0000 UTC m=+364.727166489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs") pod "model-serving-api-54d89bd5fc-9285d" (UID: "9e8488b3-e109-41ae-acb2-077daf435dfc") : secret "model-serving-api-tls" not found
Sep 06 14:03:47.208819 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.208794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7w5\" (UniqueName: \"kubernetes.io/projected/9e8488b3-e109-41ae-acb2-077daf435dfc-kube-api-access-lr7w5\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.334067 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.333976    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:47.335246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.335220    2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Sep 06 14:03:47.335347 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.335267    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Sep 06 14:03:47.425025 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.424987    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:47.429411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.429389    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:47.705503 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.705466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.707976 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.707955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e8488b3-e109-41ae-acb2-077daf435dfc-tls-certs\") pod \"model-serving-api-54d89bd5fc-9285d\" (UID: \"9e8488b3-e109-41ae-acb2-077daf435dfc\") " pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:47.912392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:47.912329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:48.005315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.005279    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:48.006259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.006240    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw"
Sep 06 14:03:48.050316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.050282    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:48.182264 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.182226    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9285d"]
Sep 06 14:03:48.184424 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:48.184391    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8488b3_e109_41ae_acb2_077daf435dfc.slice/crio-48a4fd1a01726e3e8f6ec53cc502249a6b8f7fa15e228daf92ab37920eb82718 WatchSource:0}: Error finding container 48a4fd1a01726e3e8f6ec53cc502249a6b8f7fa15e228daf92ab37920eb82718: Status 404 returned error can't find the container with id 48a4fd1a01726e3e8f6ec53cc502249a6b8f7fa15e228daf92ab37920eb82718
Sep 06 14:03:48.334330 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.334232    2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Sep 06 14:03:48.334330 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:48.334302    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Sep 06 14:03:49.011231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:49.011192    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9285d" event={"ID":"9e8488b3-e109-41ae-acb2-077daf435dfc","Type":"ContainerStarted","Data":"48a4fd1a01726e3e8f6ec53cc502249a6b8f7fa15e228daf92ab37920eb82718"}
Sep 06 14:03:49.334067 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:49.333977    2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Sep 06 14:03:49.334233 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:49.334062    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Sep 06 14:03:50.016264 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:50.016160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9285d" event={"ID":"9e8488b3-e109-41ae-acb2-077daf435dfc","Type":"ContainerStarted","Data":"6d6ffd2203d4e673372529304252abb512b764e82beef93e6885113065805b89"}
Sep 06 14:03:50.016711 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:50.016518    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy" containerID="cri-o://11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e" gracePeriod=30
Sep 06 14:03:50.031454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:50.031402    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9285d" podStartSLOduration=2.474795371 podStartE2EDuration="4.031386708s" totalImagesPullingTime="1.556591337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:50.030182972 +0000 UTC m=+367.056248148" watchObservedRunningTime="2026-09-06 14:03:50.031386708 +0000 UTC m=+367.057451880"
Sep 06 14:03:51.019837 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:51.019801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:03:53.722496 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.722460    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"]
Sep 06 14:03:53.725705 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.725687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.729006 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.728980    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 14:03:53.729143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.729026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 14:03:53.729143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.729044    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 14:03:53.729143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.729068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 14:03:53.729143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.729074    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q578l\""
Sep 06 14:03:53.729433 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.729416    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 14:03:53.735040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.735018    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"]
Sep 06 14:03:53.859387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.859349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.859600 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.859402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/221bda4b-5512-4144-a306-f1411b81f925-manager-config\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.859600 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.859504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-metrics-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.859600 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.859550    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdljg\" (UniqueName: \"kubernetes.io/projected/221bda4b-5512-4144-a306-f1411b81f925-kube-api-access-wdljg\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.960619 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.960574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-metrics-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.960619 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.960623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdljg\" (UniqueName: \"kubernetes.io/projected/221bda4b-5512-4144-a306-f1411b81f925-kube-api-access-wdljg\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.960906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.960682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.960906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.960710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/221bda4b-5512-4144-a306-f1411b81f925-manager-config\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.961435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.961414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/221bda4b-5512-4144-a306-f1411b81f925-manager-config\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.963251 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.963231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.963313 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.963300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/221bda4b-5512-4144-a306-f1411b81f925-metrics-cert\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:53.986147 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:53.986081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdljg\" (UniqueName: \"kubernetes.io/projected/221bda4b-5512-4144-a306-f1411b81f925-kube-api-access-wdljg\") pod \"lws-controller-manager-849f85686b-jlfx4\" (UID: \"221bda4b-5512-4144-a306-f1411b81f925\") " pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:54.035062 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:54.035028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:54.308833 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:54.308727    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"]
Sep 06 14:03:54.311394 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:03:54.311366    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221bda4b_5512_4144_a306_f1411b81f925.slice/crio-82ba21b58555c7de18595a8cac75cf1d71fbcf14ad6cdde06361fb790e3e50e8 WatchSource:0}: Error finding container 82ba21b58555c7de18595a8cac75cf1d71fbcf14ad6cdde06361fb790e3e50e8: Status 404 returned error can't find the container with id 82ba21b58555c7de18595a8cac75cf1d71fbcf14ad6cdde06361fb790e3e50e8
Sep 06 14:03:55.038438 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.038405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4" event={"ID":"221bda4b-5512-4144-a306-f1411b81f925","Type":"ContainerStarted","Data":"82ba21b58555c7de18595a8cac75cf1d71fbcf14ad6cdde06361fb790e3e50e8"}
Sep 06 14:03:55.310490 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.310463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:55.373129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq4fw\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373219    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373306    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373362    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket\") pod \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\" (UID: \"fab4a48b-5b15-4415-9f85-d568e51e8dfb\") "
Sep 06 14:03:55.373578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373535    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:03:55.373786 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373659    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.373848 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.373823    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:03:55.374087 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.374063    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:03:55.374809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.374138    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:03:55.374809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.374276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:03:55.376777 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.376722    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:03:55.377159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.377133    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:03:55.377159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.377431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw" (OuterVolumeSpecName: "kube-api-access-wq4fw") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "kube-api-access-wq4fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:03:55.378502 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.378473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fab4a48b-5b15-4415-9f85-d568e51e8dfb" (UID: "fab4a48b-5b15-4415-9f85-d568e51e8dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:03:55.474791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474742    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.474791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474787    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.474791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474801    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.475055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474814    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.475055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474822    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab4a48b-5b15-4415-9f85-d568e51e8dfb-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.475055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474831    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.475055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474844    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq4fw\" (UniqueName: \"kubernetes.io/projected/fab4a48b-5b15-4415-9f85-d568e51e8dfb-kube-api-access-wq4fw\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:55.475055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:55.474856    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab4a48b-5b15-4415-9f85-d568e51e8dfb-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:03:56.050812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.050750    2576 generic.go:356] "Generic (PLEG): container finished" podID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerID="11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e" exitCode=0
Sep 06 14:03:56.051277 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.050826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" event={"ID":"fab4a48b-5b15-4415-9f85-d568e51e8dfb","Type":"ContainerDied","Data":"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"}
Sep 06 14:03:56.051277 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.050860    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"
Sep 06 14:03:56.051277 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.050910    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q" event={"ID":"fab4a48b-5b15-4415-9f85-d568e51e8dfb","Type":"ContainerDied","Data":"79cd6f3925454dbbeb5830720e393e2d07a1caa086a977107c5402e2b1a481a0"}
Sep 06 14:03:56.051277 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.050942    2576 scope.go:117] "RemoveContainer" containerID="11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"
Sep 06 14:03:56.072808 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.072723    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:56.078755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.078724    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bctkc8q"]
Sep 06 14:03:56.753389 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.753242    2576 scope.go:117] "RemoveContainer" containerID="11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"
Sep 06 14:03:56.753655 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:03:56.753627    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e\": container with ID starting with 11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e not found: ID does not exist" containerID="11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"
Sep 06 14:03:56.753701 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:56.753665    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e"} err="failed to get container status \"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e\": rpc error: code = NotFound desc = could not find container \"11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e\": container with ID starting with 11cc133676d0338a41779da3703b68e297bdffac89a4f476d9cabcef11a9047e not found: ID does not exist"
Sep 06 14:03:57.061094 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:57.060988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4" event={"ID":"221bda4b-5512-4144-a306-f1411b81f925","Type":"ContainerStarted","Data":"fc415b5c9bcb9ec1ab2d8a1ab7c2d5c10cb8cbb0c154ea4e546d2358d10b2a6a"}
Sep 06 14:03:57.061541 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:57.061112    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:03:57.076581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:57.076520    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4" podStartSLOduration=1.578064576 podStartE2EDuration="4.076502823s" totalImagesPullingTime="2.498438247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:03:57.075077136 +0000 UTC m=+374.101142311" watchObservedRunningTime="2026-09-06 14:03:57.076502823 +0000 UTC m=+374.102568076"
Sep 06 14:03:57.608613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:03:57.608576    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" path="/var/lib/kubelet/pods/fab4a48b-5b15-4415-9f85-d568e51e8dfb/volumes"
Sep 06 14:04:02.028344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:04:02.028309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9285d"
Sep 06 14:04:08.067743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:04:08.067712    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-849f85686b-jlfx4"
Sep 06 14:05:56.502955 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.502918    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:05:56.503460 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.503183    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy"
Sep 06 14:05:56.503460 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.503196    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy"
Sep 06 14:05:56.503460 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.503256    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab4a48b-5b15-4415-9f85-d568e51e8dfb" containerName="istio-proxy"
Sep 06 14:05:56.506016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.505988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:05:56.508305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.508281    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 14:05:56.508451 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.508435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pjvcz\""
Sep 06 14:05:56.508983 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.508969    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 14:05:56.511811 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.511787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:05:56.641387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.641349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759f8\" (UniqueName: \"kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8\") pod \"authorino-c548b5d7d-zn8l4\" (UID: \"9c375f99-5623-4bd9-89a9-bdf2127d1de7\") " pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:05:56.742447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.742403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759f8\" (UniqueName: \"kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8\") pod \"authorino-c548b5d7d-zn8l4\" (UID: \"9c375f99-5623-4bd9-89a9-bdf2127d1de7\") " pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:05:56.749836 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.749804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759f8\" (UniqueName: \"kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8\") pod \"authorino-c548b5d7d-zn8l4\" (UID: \"9c375f99-5623-4bd9-89a9-bdf2127d1de7\") " pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:05:56.816427 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:56.816343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:05:57.082090 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:57.082020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:05:57.085104 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:05:57.085076    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c375f99_5623_4bd9_89a9_bdf2127d1de7.slice/crio-69eb9afb758918ce1c0140ecc5ecea0e6cd0906734cecdef499093affd7e606e WatchSource:0}: Error finding container 69eb9afb758918ce1c0140ecc5ecea0e6cd0906734cecdef499093affd7e606e: Status 404 returned error can't find the container with id 69eb9afb758918ce1c0140ecc5ecea0e6cd0906734cecdef499093affd7e606e
Sep 06 14:05:57.512428 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:05:57.512390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" event={"ID":"9c375f99-5623-4bd9-89a9-bdf2127d1de7","Type":"ContainerStarted","Data":"69eb9afb758918ce1c0140ecc5ecea0e6cd0906734cecdef499093affd7e606e"}
Sep 06 14:06:00.525653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:00.525617    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" event={"ID":"9c375f99-5623-4bd9-89a9-bdf2127d1de7","Type":"ContainerStarted","Data":"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"}
Sep 06 14:06:00.538459 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:00.538396    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" podStartSLOduration=2.052793201 podStartE2EDuration="4.538376898s" totalImagesPullingTime="2.485583697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:06:00.538079213 +0000 UTC m=+497.564144389" watchObservedRunningTime="2026-09-06 14:06:00.538376898 +0000 UTC m=+497.564442074"
Sep 06 14:06:26.009749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.009709    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:06:26.010182 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.009896    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" podUID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" containerName="authorino" containerID="cri-o://52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf" gracePeriod=30
Sep 06 14:06:26.248109 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.248085    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:06:26.285286 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.285187    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759f8\" (UniqueName: \"kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8\") pod \"9c375f99-5623-4bd9-89a9-bdf2127d1de7\" (UID: \"9c375f99-5623-4bd9-89a9-bdf2127d1de7\") "
Sep 06 14:06:26.287329 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.287298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8" (OuterVolumeSpecName: "kube-api-access-759f8") pod "9c375f99-5623-4bd9-89a9-bdf2127d1de7" (UID: "9c375f99-5623-4bd9-89a9-bdf2127d1de7"). InnerVolumeSpecName "kube-api-access-759f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:06:26.386418 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.386378    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759f8\" (UniqueName: \"kubernetes.io/projected/9c375f99-5623-4bd9-89a9-bdf2127d1de7-kube-api-access-759f8\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:06:26.629072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.628983    2576 generic.go:356] "Generic (PLEG): container finished" podID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" containerID="52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf" exitCode=0
Sep 06 14:06:26.629072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.629021    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-zn8l4"
Sep 06 14:06:26.629072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.629061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" event={"ID":"9c375f99-5623-4bd9-89a9-bdf2127d1de7","Type":"ContainerDied","Data":"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"}
Sep 06 14:06:26.629322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.629094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-zn8l4" event={"ID":"9c375f99-5623-4bd9-89a9-bdf2127d1de7","Type":"ContainerDied","Data":"69eb9afb758918ce1c0140ecc5ecea0e6cd0906734cecdef499093affd7e606e"}
Sep 06 14:06:26.629322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.629115    2576 scope.go:117] "RemoveContainer" containerID="52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"
Sep 06 14:06:26.637073 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.637056    2576 scope.go:117] "RemoveContainer" containerID="52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"
Sep 06 14:06:26.637343 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:06:26.637318    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf\": container with ID starting with 52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf not found: ID does not exist" containerID="52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"
Sep 06 14:06:26.637413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.637355    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf"} err="failed to get container status \"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf\": rpc error: code = NotFound desc = could not find container \"52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf\": container with ID starting with 52b4b16ad671d3494e1a9ca938f8ab6f794981a41134bdb293bd22bb9a2083bf not found: ID does not exist"
Sep 06 14:06:26.647816 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.647789    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:06:26.653473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:26.653448    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-zn8l4"]
Sep 06 14:06:27.612457 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:06:27.612425    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" path="/var/lib/kubelet/pods/9c375f99-5623-4bd9-89a9-bdf2127d1de7/volumes"
Sep 06 14:07:43.427175 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:07:43.427148    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:07:43.572054 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:07:43.572027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:08:00.995387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:00.995350    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"]
Sep 06 14:08:00.995869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:00.995647    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" containerName="authorino"
Sep 06 14:08:00.995869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:00.995659    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" containerName="authorino"
Sep 06 14:08:00.995869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:00.995712    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c375f99-5623-4bd9-89a9-bdf2127d1de7" containerName="authorino"
Sep 06 14:08:00.998603 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:00.998587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.000863 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.000842    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 14:08:01.001111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.001094    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 14:08:01.001879 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.001862    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 06 14:08:01.001965 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.001917    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4dz5\""
Sep 06 14:08:01.008458 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.008435    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"]
Sep 06 14:08:01.024407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.024614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d79ee-2721-4abf-80e6-28b4baebfecc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.024614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.024614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.024799 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.024799 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.024748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslrg\" (UniqueName: \"kubernetes.io/projected/f18d79ee-2721-4abf-80e6-28b4baebfecc-kube-api-access-kslrg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.125733 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.125692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.125733 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.125729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d79ee-2721-4abf-80e6-28b4baebfecc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.125966 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.125748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.125966 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.125888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.126044 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.126009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.126080 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.126041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslrg\" (UniqueName: \"kubernetes.io/projected/f18d79ee-2721-4abf-80e6-28b4baebfecc-kube-api-access-kslrg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.126159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.126141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.126298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.126278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.126367 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.126319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.128127 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.128103    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18d79ee-2721-4abf-80e6-28b4baebfecc-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.128324 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.128305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18d79ee-2721-4abf-80e6-28b4baebfecc-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.132958 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.132935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslrg\" (UniqueName: \"kubernetes.io/projected/f18d79ee-2721-4abf-80e6-28b4baebfecc-kube-api-access-kslrg\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lghxs\" (UID: \"f18d79ee-2721-4abf-80e6-28b4baebfecc\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.309708 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.309609    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:01.577344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.577317    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"]
Sep 06 14:08:01.580094 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:08:01.580064    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18d79ee_2721_4abf_80e6_28b4baebfecc.slice/crio-389b45b7597d1934dce697588e2f9371f3af6aed971914d5db75801e7a714f56 WatchSource:0}: Error finding container 389b45b7597d1934dce697588e2f9371f3af6aed971914d5db75801e7a714f56: Status 404 returned error can't find the container with id 389b45b7597d1934dce697588e2f9371f3af6aed971914d5db75801e7a714f56
Sep 06 14:08:01.983949 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:01.983900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs" event={"ID":"f18d79ee-2721-4abf-80e6-28b4baebfecc","Type":"ContainerStarted","Data":"389b45b7597d1934dce697588e2f9371f3af6aed971914d5db75801e7a714f56"}
Sep 06 14:08:02.989664 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:02.989627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs" event={"ID":"f18d79ee-2721-4abf-80e6-28b4baebfecc","Type":"ContainerStarted","Data":"e8e670a89c917917d87c4e0b4b5dca8a9518abb39a62d5addcc76473e99de337"}
Sep 06 14:08:03.005220 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:03.005165    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs" podStartSLOduration=2.060673056 podStartE2EDuration="3.005150293s" totalImagesPullingTime="944.477237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:08:03.003562051 +0000 UTC m=+620.029627236" watchObservedRunningTime="2026-09-06 14:08:03.005150293 +0000 UTC m=+620.031215468"
Sep 06 14:08:07.088200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.088084    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"]
Sep 06 14:08:07.091461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.091437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.093496 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.093472    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 06 14:08:07.101234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.101210    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"]
Sep 06 14:08:07.173888 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.173851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.174059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.173899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.174059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.173966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.174059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.174007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtwz\" (UniqueName: \"kubernetes.io/projected/6919c6fe-8749-4d52-a7de-d5c298eb5b40-kube-api-access-2gtwz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.174209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.174052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.174209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.174085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275340 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtwz\" (UniqueName: \"kubernetes.io/projected/6919c6fe-8749-4d52-a7de-d5c298eb5b40-kube-api-access-2gtwz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275623    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275820 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275876 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.275942 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.275925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.277713 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.277683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6919c6fe-8749-4d52-a7de-d5c298eb5b40-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.278007 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.277986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6919c6fe-8749-4d52-a7de-d5c298eb5b40-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.282210 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.282190    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtwz\" (UniqueName: \"kubernetes.io/projected/6919c6fe-8749-4d52-a7de-d5c298eb5b40-kube-api-access-2gtwz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg\" (UID: \"6919c6fe-8749-4d52-a7de-d5c298eb5b40\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.402011 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.401910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:07.670628 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:07.670600    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"]
Sep 06 14:08:07.673523 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:08:07.673495    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6919c6fe_8749_4d52_a7de_d5c298eb5b40.slice/crio-1f3d8676a99a45dbb5311925e935031a2758cbb0bee807fdd7d0db1ec9dbb5f0 WatchSource:0}: Error finding container 1f3d8676a99a45dbb5311925e935031a2758cbb0bee807fdd7d0db1ec9dbb5f0: Status 404 returned error can't find the container with id 1f3d8676a99a45dbb5311925e935031a2758cbb0bee807fdd7d0db1ec9dbb5f0
Sep 06 14:08:08.011090 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:08.011002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg" event={"ID":"6919c6fe-8749-4d52-a7de-d5c298eb5b40","Type":"ContainerStarted","Data":"67b58e48baa04019828fea2c5c18bd67614059ba6fc11ef30939cf58bd9a860d"}
Sep 06 14:08:08.011090 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:08.011070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg" event={"ID":"6919c6fe-8749-4d52-a7de-d5c298eb5b40","Type":"ContainerStarted","Data":"1f3d8676a99a45dbb5311925e935031a2758cbb0bee807fdd7d0db1ec9dbb5f0"}
Sep 06 14:08:08.029637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:08.029587    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg" podStartSLOduration=0.869065195 podStartE2EDuration="1.029569754s" totalImagesPullingTime="160.504559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:08:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:08:08.028721718 +0000 UTC m=+625.054786907" watchObservedRunningTime="2026-09-06 14:08:08.029569754 +0000 UTC m=+625.055634928"
Sep 06 14:08:11.310170 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:11.310130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:11.310530 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:11.310211    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:11.322870 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:11.322846    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:12.038285 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:12.038257    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lghxs"
Sep 06 14:08:17.402803 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:17.402738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:17.405259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:17.402838    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:17.415202 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:17.415174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:08:18.061467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:08:18.061439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg"
Sep 06 14:09:11.589713 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.589676    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"]
Sep 06 14:09:11.593018 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.593000    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.595088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.595064    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 06 14:09:11.602272 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.602239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"]
Sep 06 14:09:11.705463 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.705463 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ab3082-1fb0-43c5-8ff1-18881db755b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.705685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg5jn\" (UniqueName: \"kubernetes.io/projected/37ab3082-1fb0-43c5-8ff1-18881db755b7-kube-api-access-zg5jn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.705685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.705685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.705830 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.705725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ab3082-1fb0-43c5-8ff1-18881db755b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg5jn\" (UniqueName: \"kubernetes.io/projected/37ab3082-1fb0-43c5-8ff1-18881db755b7-kube-api-access-zg5jn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806741 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806741 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.806884 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.806864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.807039 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.807020    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.807101 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.807062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.808575 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.808552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37ab3082-1fb0-43c5-8ff1-18881db755b7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.808890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.808873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37ab3082-1fb0-43c5-8ff1-18881db755b7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.813676 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.813648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg5jn\" (UniqueName: \"kubernetes.io/projected/37ab3082-1fb0-43c5-8ff1-18881db755b7-kube-api-access-zg5jn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp\" (UID: \"37ab3082-1fb0-43c5-8ff1-18881db755b7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:11.903036 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:11.902940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:12.173589 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:12.173563    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"]
Sep 06 14:09:12.176020 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:09:12.175991    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ab3082_1fb0_43c5_8ff1_18881db755b7.slice/crio-dbebf003b8233091b4d18db4e1b32cc404dba153c6a2aed90a1a26821df5c471 WatchSource:0}: Error finding container dbebf003b8233091b4d18db4e1b32cc404dba153c6a2aed90a1a26821df5c471: Status 404 returned error can't find the container with id dbebf003b8233091b4d18db4e1b32cc404dba153c6a2aed90a1a26821df5c471
Sep 06 14:09:12.177251 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:12.177234    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 14:09:12.267754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:12.267712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp" event={"ID":"37ab3082-1fb0-43c5-8ff1-18881db755b7","Type":"ContainerStarted","Data":"dbebf003b8233091b4d18db4e1b32cc404dba153c6a2aed90a1a26821df5c471"}
Sep 06 14:09:13.273123 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:13.273090    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp" event={"ID":"37ab3082-1fb0-43c5-8ff1-18881db755b7","Type":"ContainerStarted","Data":"adf6da3a7c418c1777d9bca7ada8283073b6b7f7e8905beaa7819c1d5b483b07"}
Sep 06 14:09:13.287426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:13.287364    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp" podStartSLOduration=2.115734782 podStartE2EDuration="2.287344853s" totalImagesPullingTime="171.610071ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:09:13.286927757 +0000 UTC m=+690.312992932" watchObservedRunningTime="2026-09-06 14:09:13.287344853 +0000 UTC m=+690.313410029"
Sep 06 14:09:21.903966 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:21.903928    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:21.904377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:21.904149    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:21.916671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:21.916636    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:09:22.320307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:09:22.320280    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp"
Sep 06 14:10:25.625712 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.625669    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:10:25.629324 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.629306    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.631515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.631493    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dfd6b-openshift-default-dockercfg-6x5t2\""
Sep 06 14:10:25.640680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.640655    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:10:25.725219 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725219 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725301    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgm7\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.725440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.725418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.826659 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.826659 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826716    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826812    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgm7\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.826883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.827295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.827348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827540 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.827504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.827547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.827739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.827719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.829085 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.829057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.829322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.829304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.833851 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.833824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.833985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.833966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgm7\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7\") pod \"e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") " pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:25.941814 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:25.941777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:26.216114 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.216085    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:10:26.218126 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:10:26.218098    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c5e75f_1c79_469b_ab3b_b1b7e8359deb.slice/crio-47cd5f13cfa69088f48931ead8c236dea0890b47336f3800d1ce10f94ccec424 WatchSource:0}: Error finding container 47cd5f13cfa69088f48931ead8c236dea0890b47336f3800d1ce10f94ccec424: Status 404 returned error can't find the container with id 47cd5f13cfa69088f48931ead8c236dea0890b47336f3800d1ce10f94ccec424
Sep 06 14:10:26.220669 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.220634    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:26.220739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.220700    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:26.220739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.220730    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:26.570343 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.570250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerStarted","Data":"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"}
Sep 06 14:10:26.570343 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.570305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerStarted","Data":"47cd5f13cfa69088f48931ead8c236dea0890b47336f3800d1ce10f94ccec424"}
Sep 06 14:10:26.590553 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.590489    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podStartSLOduration=1.590473815 podStartE2EDuration="1.590473815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:26.588723676 +0000 UTC m=+763.614788854" watchObservedRunningTime="2026-09-06 14:10:26.590473815 +0000 UTC m=+763.616538989"
Sep 06 14:10:26.941871 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:26.941840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:27.948038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:27.948007    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:28.579968 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:28.579941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:28.581084 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:28.581060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:29.808686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:29.808653    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7554-140212513809712"]
Sep 06 14:10:29.811942 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:29.811924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:29.817782 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:29.815250    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\""
Sep 06 14:10:29.818685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:29.818655    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"]
Sep 06 14:10:29.963184 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:29.963143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzsj\" (UniqueName: \"kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj\") pod \"test-curl-7554-140212513809712\" (UID: \"bee97713-ddbf-4de5-a8e6-585e3d155975\") " pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:30.063781 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:30.063663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzsj\" (UniqueName: \"kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj\") pod \"test-curl-7554-140212513809712\" (UID: \"bee97713-ddbf-4de5-a8e6-585e3d155975\") " pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:30.073213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:30.073184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzsj\" (UniqueName: \"kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj\") pod \"test-curl-7554-140212513809712\" (UID: \"bee97713-ddbf-4de5-a8e6-585e3d155975\") " pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:30.129881 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:30.129850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:30.404663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:30.404640    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"]
Sep 06 14:10:30.410101 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:10:30.410065    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee97713_ddbf_4de5_a8e6_585e3d155975.slice/crio-b12940bf1e002357604c90a0e73febc0663dc92c6b89060a67e49987f6f93674 WatchSource:0}: Error finding container b12940bf1e002357604c90a0e73febc0663dc92c6b89060a67e49987f6f93674: Status 404 returned error can't find the container with id b12940bf1e002357604c90a0e73febc0663dc92c6b89060a67e49987f6f93674
Sep 06 14:10:30.589839 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:30.589802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-140212513809712" event={"ID":"bee97713-ddbf-4de5-a8e6-585e3d155975","Type":"ContainerStarted","Data":"b12940bf1e002357604c90a0e73febc0663dc92c6b89060a67e49987f6f93674"}
Sep 06 14:10:33.478995 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.478956    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:10:33.482489 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.482467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.484775 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.484734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-84753aa3-openshift-default-dockercfg-gt27n\""
Sep 06 14:10:33.494360 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.494333    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:10:33.597837 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.597805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.597844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.597864    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8s6d\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.597934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598154 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.598035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598154 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.598071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598154 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.598094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598154 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.598111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.598323 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.598204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.606708 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.606680    2576 generic.go:356] "Generic (PLEG): container finished" podID="bee97713-ddbf-4de5-a8e6-585e3d155975" containerID="66a743c0f0e7882e22a7e77f57bddfb4c4e17a99671493f24cb0b7a5321e8ce5" exitCode=0
Sep 06 14:10:33.609509 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.609479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7554-140212513809712" event={"ID":"bee97713-ddbf-4de5-a8e6-585e3d155975","Type":"ContainerDied","Data":"66a743c0f0e7882e22a7e77f57bddfb4c4e17a99671493f24cb0b7a5321e8ce5"}
Sep 06 14:10:33.699311 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699311 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8s6d\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.699795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699731    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.700037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.699797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.700037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.700000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.700376 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.700358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.701804 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.701777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.702067 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.702047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.706522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.706499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8s6d\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.706522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.706514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token\") pod \"e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") " pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:33.794405 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:33.794326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:34.069997 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.069916    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:10:34.072846 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:10:34.072819    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9233b4a_d3b9_482d_b4ef_73cc4135ef84.slice/crio-11172ed59211a3dda86485785d1b34792e9ab2921dc00a7433a114c7af725276 WatchSource:0}: Error finding container 11172ed59211a3dda86485785d1b34792e9ab2921dc00a7433a114c7af725276: Status 404 returned error can't find the container with id 11172ed59211a3dda86485785d1b34792e9ab2921dc00a7433a114c7af725276
Sep 06 14:10:34.075186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.075143    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:34.075265 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.075237    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:34.075305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.075278    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:34.612264 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.612225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerStarted","Data":"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"}
Sep 06 14:10:34.612708 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.612276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerStarted","Data":"11172ed59211a3dda86485785d1b34792e9ab2921dc00a7433a114c7af725276"}
Sep 06 14:10:34.632217 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.632156    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podStartSLOduration=1.6321380479999998 podStartE2EDuration="1.632138048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:34.629994729 +0000 UTC m=+771.656059905" watchObservedRunningTime="2026-09-06 14:10:34.632138048 +0000 UTC m=+771.658203226"
Sep 06 14:10:34.738581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.738555    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:34.771217 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.771186    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7554-140212513809712_bee97713-ddbf-4de5-a8e6-585e3d155975/test-curl-7554-140212513809712/0.log"
Sep 06 14:10:34.795249 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.795204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:34.800149 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.800122    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"]
Sep 06 14:10:34.803476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.803453    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7554-140212513809712"]
Sep 06 14:10:34.911588 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.911485    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njzsj\" (UniqueName: \"kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj\") pod \"bee97713-ddbf-4de5-a8e6-585e3d155975\" (UID: \"bee97713-ddbf-4de5-a8e6-585e3d155975\") "
Sep 06 14:10:34.913653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:34.913626    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj" (OuterVolumeSpecName: "kube-api-access-njzsj") pod "bee97713-ddbf-4de5-a8e6-585e3d155975" (UID: "bee97713-ddbf-4de5-a8e6-585e3d155975"). InnerVolumeSpecName "kube-api-access-njzsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:10:35.012393 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.012351    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzsj\" (UniqueName: \"kubernetes.io/projected/bee97713-ddbf-4de5-a8e6-585e3d155975-kube-api-access-njzsj\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:10:35.535749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.535711    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:10:35.536106 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.536089    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee97713-ddbf-4de5-a8e6-585e3d155975" containerName="test-curl-7554-140212513809712"
Sep 06 14:10:35.536170 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.536110    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee97713-ddbf-4de5-a8e6-585e3d155975" containerName="test-curl-7554-140212513809712"
Sep 06 14:10:35.536223 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.536174    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee97713-ddbf-4de5-a8e6-585e3d155975" containerName="test-curl-7554-140212513809712"
Sep 06 14:10:35.539018 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.538999    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.541136 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.541115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-1482ae-openshift-default-dockercfg-z92qt\""
Sep 06 14:10:35.550554 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.550528    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:10:35.609805 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.609746    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee97713-ddbf-4de5-a8e6-585e3d155975" path="/var/lib/kubelet/pods/bee97713-ddbf-4de5-a8e6-585e3d155975/volumes"
Sep 06 14:10:35.616869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.616842    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7554-140212513809712"
Sep 06 14:10:35.616869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.616863    2576 scope.go:117] "RemoveContainer" containerID="66a743c0f0e7882e22a7e77f57bddfb4c4e17a99671493f24cb0b7a5321e8ce5"
Sep 06 14:10:35.717978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.717942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhnl\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.717988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718245 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718282 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718320 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718320 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718444 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.718444 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.718430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.800814 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.800724    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:35.819095 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhnl\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819502 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819502 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819502 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819635 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819729 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819695    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819729 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.819961 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.819933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.820040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.820003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.820367 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.820342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.821986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.821964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.822502 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.822481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.826002 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.825965    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhnl\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.826134 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.826114    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token\") pod \"e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") " pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:35.849412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:35.849375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:36.128058 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.128031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:10:36.130022 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:10:36.129993    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde04245_e65d_41b6_9dbb_4cbea2701f40.slice/crio-50d08b7a7a33dfcd787b6d3eb9c6f21d19d38e357c2892af95de6287d3d95424 WatchSource:0}: Error finding container 50d08b7a7a33dfcd787b6d3eb9c6f21d19d38e357c2892af95de6287d3d95424: Status 404 returned error can't find the container with id 50d08b7a7a33dfcd787b6d3eb9c6f21d19d38e357c2892af95de6287d3d95424
Sep 06 14:10:36.132238 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.132207    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:36.132329 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.132281    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:36.132390 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.132329    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:36.622867 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.622829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerStarted","Data":"54d5bb4dda45b0e84d3b0d1ecbee1a8988bc68ddb81dd1431b226f35c44c9531"}
Sep 06 14:10:36.623340 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.622887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerStarted","Data":"50d08b7a7a33dfcd787b6d3eb9c6f21d19d38e357c2892af95de6287d3d95424"}
Sep 06 14:10:36.623340 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.623060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:36.624118 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.624100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:36.643378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.643323    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podStartSLOduration=1.6433073839999999 podStartE2EDuration="1.643307384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:36.641057699 +0000 UTC m=+773.667122874" watchObservedRunningTime="2026-09-06 14:10:36.643307384 +0000 UTC m=+773.669372624"
Sep 06 14:10:36.850434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:36.850395    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:37.855601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:37.855572    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:38.636800 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:38.636752    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:38.637837 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:38.637816    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:10:44.581120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:44.581071    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:10:44.581490 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:44.581155    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:10:46.670461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.670426    2576 generic.go:356] "Generic (PLEG): container finished" podID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerID="ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7" exitCode=0
Sep 06 14:10:46.671020 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.670505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerDied","Data":"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"}
Sep 06 14:10:46.671091 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.671041    2576 scope.go:117] "RemoveContainer" containerID="ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"
Sep 06 14:10:46.672140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.672110    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:46.672231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.672179    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:46.672231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:46.672209    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:47.676675 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:47.676641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerStarted","Data":"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"}
Sep 06 14:10:47.942748 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:47.942643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:47.943161 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:47.943044    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Sep 06 14:10:47.943344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:47.943320    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Sep 06 14:10:48.576244 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.576213    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:10:48.583215 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.583191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.585674 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.585653    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-8318d32e-kserve-self-signed-certs\""
Sep 06 14:10:48.587024 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.587002    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:10:48.632315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.632247    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.632694 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.632670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.632908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.632889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.633103 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.633087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.633303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.633287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.633465 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.633448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrmxw\" (UniqueName: \"kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.734491 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.734429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrmxw\" (UniqueName: \"kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.735579 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.735551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.736570 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.736551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.737084 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.737067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.737636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.737617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.738101 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.737509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.739361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.738008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.739488 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.738164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.740322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.738412    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.740482 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.739106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.740605 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.739744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.747480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.747451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrmxw\" (UniqueName: \"kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw\") pod \"e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") " pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.893603 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.893504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:48.945798 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.944567    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Sep 06 14:10:48.945798 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:48.944640    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Sep 06 14:10:49.219024 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.218984    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:10:49.691904 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.691864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" event={"ID":"7fbf8011-000b-48aa-a677-50f02bfe041f","Type":"ContainerStarted","Data":"0eb492f21aa68a00670c9c27120b06390abed5eb2115d52aff720643244322e8"}
Sep 06 14:10:49.692098 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.691911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" event={"ID":"7fbf8011-000b-48aa-a677-50f02bfe041f","Type":"ContainerStarted","Data":"c9087cbec3601c3a0ce0dacd28e75220c6c0c5d43241bdce2e92473018a3bf08"}
Sep 06 14:10:49.719787 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.710983    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" podStartSLOduration=1.71095708 podStartE2EDuration="1.71095708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:49.710257646 +0000 UTC m=+786.736322820" watchObservedRunningTime="2026-09-06 14:10:49.71095708 +0000 UTC m=+786.737022257"
Sep 06 14:10:49.942396 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.942304    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Sep 06 14:10:49.942396 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:49.942375    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Sep 06 14:10:50.943058 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:50.943020    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Sep 06 14:10:50.943478 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:50.943076    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Sep 06 14:10:51.943029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:51.942979    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Sep 06 14:10:51.943310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:51.943276    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Sep 06 14:10:52.666026 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:52.665979    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:10:52.666238 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:52.666060    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:10:53.194562 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.194526    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:53.195102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.194941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:53.197127 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.197097    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:10:53.715788 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.713709    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerID="98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d" exitCode=0
Sep 06 14:10:53.715788 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.714540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerDied","Data":"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"}
Sep 06 14:10:53.715788 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.714935    2576 scope.go:117] "RemoveContainer" containerID="98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"
Sep 06 14:10:53.716473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.716125    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:53.716473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.716207    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:53.716473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.716252    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:53.972778 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.972680    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:10:53.975281 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.975259    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:53.978025 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.978002    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ec585-openshift-default-dockercfg-nq9fb\""
Sep 06 14:10:53.993108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:53.991694    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:10:54.099346 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfqv\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099346 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.099744 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.099674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.200609 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.200609 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfqv\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.200997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.201913 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.201882    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.203141 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.203123    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.203697 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.203673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.207794 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.207751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.208088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.208067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfqv\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv\") pod \"e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") " pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.288408 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.288301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:54.605055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.605027    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:10:54.607444 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:10:54.607417    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09a9b77_a411_4f5d_8915_03a7be0f3adc.slice/crio-b7cb2ce91859db0eacc4f2fef252680630dd5367ecfe6d9a395dc017d6f6707b WatchSource:0}: Error finding container b7cb2ce91859db0eacc4f2fef252680630dd5367ecfe6d9a395dc017d6f6707b: Status 404 returned error can't find the container with id b7cb2ce91859db0eacc4f2fef252680630dd5367ecfe6d9a395dc017d6f6707b
Sep 06 14:10:54.609804 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.609733    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:54.609874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.609820    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:54.609874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.609848    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:10:54.722739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.722691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerStarted","Data":"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"}
Sep 06 14:10:54.722945 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.722783    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerStarted","Data":"b7cb2ce91859db0eacc4f2fef252680630dd5367ecfe6d9a395dc017d6f6707b"}
Sep 06 14:10:54.728337 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.728285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerStarted","Data":"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"}
Sep 06 14:10:54.740210 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.740134    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podStartSLOduration=1.740111754 podStartE2EDuration="1.740111754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:10:54.739507209 +0000 UTC m=+791.765572387" watchObservedRunningTime="2026-09-06 14:10:54.740111754 +0000 UTC m=+791.766176927"
Sep 06 14:10:54.795145 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.795111    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:10:54.795340 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.795311    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:54.795456 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:54.795427    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:10:55.289092 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.289058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:55.294303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.294276    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:55.735594 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.735560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:55.747336 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.747296    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:10:55.795725 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.795694    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:55.795906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:55.795750    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:10:56.795662 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:56.795631    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:56.796244 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:56.795855    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:10:57.795501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:57.795466    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:57.795687 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:57.795524    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:10:58.795215 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:58.795176    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:58.795397 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:58.795268    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:10:58.894368 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:58.894329    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:58.894831 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:58.894390    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:58.907551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:58.907518    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:59.765866 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:59.765832    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:10:59.795028 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:59.794986    2576 patch_prober.go:28] interesting pod/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Sep 06 14:10:59.795209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:10:59.795058    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Sep 06 14:11:00.799215 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:00.799185    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:11:00.799641 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:00.799487    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:11:00.800423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:00.800402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:11:01.349586 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.349515    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"]
Sep 06 14:11:01.352627 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.352610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:01.354753 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.354727    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:11:01.354849 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.354754    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:11:01.360699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.360642    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"]
Sep 06 14:11:01.493581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.493544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2\" (UID: \"2219a47f-f5a9-425b-93bc-99c2a66addf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:01.594182 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.594145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2\" (UID: \"2219a47f-f5a9-425b-93bc-99c2a66addf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:01.594819 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.594799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2\" (UID: \"2219a47f-f5a9-425b-93bc-99c2a66addf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:01.664214 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.664174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:01.766161 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.766080    2576 generic.go:356] "Generic (PLEG): container finished" podID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerID="54d5bb4dda45b0e84d3b0d1ecbee1a8988bc68ddb81dd1431b226f35c44c9531" exitCode=0
Sep 06 14:11:01.766300 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.766152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerDied","Data":"54d5bb4dda45b0e84d3b0d1ecbee1a8988bc68ddb81dd1431b226f35c44c9531"}
Sep 06 14:11:01.766605 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.766589    2576 scope.go:117] "RemoveContainer" containerID="54d5bb4dda45b0e84d3b0d1ecbee1a8988bc68ddb81dd1431b226f35c44c9531"
Sep 06 14:11:01.767699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.767675    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:01.767771 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.767745    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:01.767806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.767791    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:01.939855 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:01.939818    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"]
Sep 06 14:11:01.942980 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:11:01.942941    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2219a47f_f5a9_425b_93bc_99c2a66addf6.slice/crio-e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1 WatchSource:0}: Error finding container e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1: Status 404 returned error can't find the container with id e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1
Sep 06 14:11:02.772658 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:02.772622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerStarted","Data":"04b8076fdeafe534480b7b28bedd6045c07d03ca2415a955508c09baeff58801"}
Sep 06 14:11:02.773974 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:02.773950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2" event={"ID":"2219a47f-f5a9-425b-93bc-99c2a66addf6","Type":"ContainerStarted","Data":"e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1"}
Sep 06 14:11:02.850120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:02.850085    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:11:02.850457 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:02.850429    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:02.850555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:02.850483    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:03.850744 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:03.850699    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:03.851190 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:03.850782    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:04.850035 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:04.850002    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:04.850198 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:04.850054    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:05.789637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:05.789516    2576 generic.go:356] "Generic (PLEG): container finished" podID="2219a47f-f5a9-425b-93bc-99c2a66addf6" containerID="2fc5f4b8fd6369079bc52bad771f27dd136d49203146176e6efd44713eacf699" exitCode=0
Sep 06 14:11:05.789637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:05.789597    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2" event={"ID":"2219a47f-f5a9-425b-93bc-99c2a66addf6","Type":"ContainerDied","Data":"2fc5f4b8fd6369079bc52bad771f27dd136d49203146176e6efd44713eacf699"}
Sep 06 14:11:05.850191 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:05.850154    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:05.850362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:05.850225    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:06.850294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:06.850257    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:06.850629 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:06.850311    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:06.959801 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:06.959777    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:06.967754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:06.967732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca\") pod \"2219a47f-f5a9-425b-93bc-99c2a66addf6\" (UID: \"2219a47f-f5a9-425b-93bc-99c2a66addf6\") "
Sep 06 14:11:06.968072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:06.968049    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2219a47f-f5a9-425b-93bc-99c2a66addf6" (UID: "2219a47f-f5a9-425b-93bc-99c2a66addf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:11:07.068834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.068719    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2219a47f-f5a9-425b-93bc-99c2a66addf6-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:07.799709 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.799681    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"
Sep 06 14:11:07.799891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.799716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2" event={"ID":"2219a47f-f5a9-425b-93bc-99c2a66addf6","Type":"ContainerDied","Data":"e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1"}
Sep 06 14:11:07.799891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.799775    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f42d39a41d30e50e9089c1747f7c45bb5b31a1083809af684352dc9f6143a1"
Sep 06 14:11:07.850627 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.850593    2576 patch_prober.go:28] interesting pod/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Sep 06 14:11:07.851036 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:07.850653    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Sep 06 14:11:08.636708 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:08.636660    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:11:08.854200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:08.854170    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:11:08.855079 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:08.855058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:11:18.014775 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.014727    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"]
Sep 06 14:11:18.017295 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.017271    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-84753aa3-n4tr2"]
Sep 06 14:11:18.142904 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.142867    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:11:18.143188 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.143148    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy" containerID="cri-o://c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92" gracePeriod=30
Sep 06 14:11:18.857669 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.857583    2576 generic.go:356] "Generic (PLEG): container finished" podID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerID="62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847" exitCode=0
Sep 06 14:11:18.857669 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.857660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerDied","Data":"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"}
Sep 06 14:11:18.858145 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.858128    2576 scope.go:117] "RemoveContainer" containerID="62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"
Sep 06 14:11:18.859252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.859220    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:18.859382 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.859314    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:18.859382 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:18.859362    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:19.614545 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:19.614506    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2219a47f-f5a9-425b-93bc-99c2a66addf6" path="/var/lib/kubelet/pods/2219a47f-f5a9-425b-93bc-99c2a66addf6/volumes"
Sep 06 14:11:19.864827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:19.864732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerStarted","Data":"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"}
Sep 06 14:11:20.288788 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:20.288727    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:11:20.289012 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:20.288988    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 06 14:11:20.289064 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:20.289050    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 06 14:11:21.288835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:21.288794    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 06 14:11:21.289247 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:21.288857    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 06 14:11:22.289451 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:22.289414    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 06 14:11:22.289842 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:22.289482    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 06 14:11:23.289345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.289308    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 06 14:11:23.289509 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.289376    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 06 14:11:23.402725 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.402700    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:11:23.543356 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543314    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543356 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543360    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543417    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8s6d\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543494    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543554    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543617    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543980 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543638    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:11:23.543980 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543651    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token\") pod \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\" (UID: \"f9233b4a-d3b9-482d-b4ef-73cc4135ef84\") "
Sep 06 14:11:23.543980 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.543903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:11:23.544133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.544045    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.544133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.544055    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:11:23.544133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.544066    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:11:23.544133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.544065    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.544133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.544088    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:11:23.545870 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.545840    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:11:23.545986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.545931    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:11:23.545986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.545954    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d" (OuterVolumeSpecName: "kube-api-access-z8s6d") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "kube-api-access-z8s6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:11:23.546061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.546002    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9233b4a-d3b9-482d-b4ef-73cc4135ef84" (UID: "f9233b4a-d3b9-482d-b4ef-73cc4135ef84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:11:23.645351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645316    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8s6d\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-kube-api-access-z8s6d\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645345    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645355    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645563 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645363    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645563 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645372    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645563 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645379    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.645563 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.645387    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9233b4a-d3b9-482d-b4ef-73cc4135ef84-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:23.884318 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.884222    2576 generic.go:356] "Generic (PLEG): container finished" podID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerID="c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92" exitCode=0
Sep 06 14:11:23.884318 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.884274    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"
Sep 06 14:11:23.884318 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.884272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerDied","Data":"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"}
Sep 06 14:11:23.884578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.884332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln" event={"ID":"f9233b4a-d3b9-482d-b4ef-73cc4135ef84","Type":"ContainerDied","Data":"11172ed59211a3dda86485785d1b34792e9ab2921dc00a7433a114c7af725276"}
Sep 06 14:11:23.884578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.884358    2576 scope.go:117] "RemoveContainer" containerID="c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"
Sep 06 14:11:23.892920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.892900    2576 scope.go:117] "RemoveContainer" containerID="98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"
Sep 06 14:11:23.899645 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.899619    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:11:23.901422 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.901353    2576 scope.go:117] "RemoveContainer" containerID="c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"
Sep 06 14:11:23.901793 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:11:23.901751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92\": container with ID starting with c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92 not found: ID does not exist" containerID="c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"
Sep 06 14:11:23.901858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.901804    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92"} err="failed to get container status \"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92\": rpc error: code = NotFound desc = could not find container \"c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92\": container with ID starting with c539ef1652832065ff44599ce058318a46f94593888c6242c029abf19fcb6c92 not found: ID does not exist"
Sep 06 14:11:23.901858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.901824    2576 scope.go:117] "RemoveContainer" containerID="98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"
Sep 06 14:11:23.902167 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:11:23.902146    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d\": container with ID starting with 98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d not found: ID does not exist" containerID="98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"
Sep 06 14:11:23.902228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.902175    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d"} err="failed to get container status \"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d\": rpc error: code = NotFound desc = could not find container \"98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d\": container with ID starting with 98053d7e1bbcb78667905eeb863f27d640eae58ab7dbd292fb5feb930bd63a9d not found: ID does not exist"
Sep 06 14:11:23.902818 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:23.902800    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-84753aa3-openshift-default-67f9c584d8-x2wln"]
Sep 06 14:11:24.289295 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:24.289254    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 06 14:11:24.289508 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:24.289329    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 06 14:11:25.338155 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:25.338121    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:11:25.338749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:25.338730    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:11:25.339464 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:25.339441    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:11:25.609918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:25.609827    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" path="/var/lib/kubelet/pods/f9233b4a-d3b9-482d-b4ef-73cc4135ef84/volumes"
Sep 06 14:11:42.791878 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.791842    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792156    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2219a47f-f5a9-425b-93bc-99c2a66addf6" containerName="revoke-keys"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792169    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2219a47f-f5a9-425b-93bc-99c2a66addf6" containerName="revoke-keys"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792179    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792184    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792203    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792208    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792263    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792273    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2219a47f-f5a9-425b-93bc-99c2a66addf6" containerName="revoke-keys"
Sep 06 14:11:42.792307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.792280    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9233b4a-d3b9-482d-b4ef-73cc4135ef84" containerName="istio-proxy"
Sep 06 14:11:42.795209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.795186    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.797448 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.797424    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-8318d32e-kserve-self-signed-certs\""
Sep 06 14:11:42.802413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.802391    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:11:42.812166 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812137    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.812301 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.812301 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.812301 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.812301 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.812433 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.812318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4rz\" (UniqueName: \"kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.912903 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.912857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.912922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.912956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.912992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.913029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4rz\" (UniqueName: \"kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.913093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913369 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.913347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913428 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.913392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.913482 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.913451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.915187 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.915165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.915432 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.915414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:42.920084 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:42.920062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4rz\" (UniqueName: \"kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz\") pod \"e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") " pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:43.106168 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:43.106069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:43.382274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:43.382115    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:11:43.384868 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:11:43.384840    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bcb6cf9_e1f4_43ef_87b5_490a8d5d2f33.slice/crio-f105eb35a97f4a35ec1b0786662cef95284fdd16a4591c6f5411af4abb072f56 WatchSource:0}: Error finding container f105eb35a97f4a35ec1b0786662cef95284fdd16a4591c6f5411af4abb072f56: Status 404 returned error can't find the container with id f105eb35a97f4a35ec1b0786662cef95284fdd16a4591c6f5411af4abb072f56
Sep 06 14:11:43.976811 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:43.976752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" event={"ID":"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33","Type":"ContainerStarted","Data":"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"}
Sep 06 14:11:43.976811 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:43.976816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" event={"ID":"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33","Type":"ContainerStarted","Data":"f105eb35a97f4a35ec1b0786662cef95284fdd16a4591c6f5411af4abb072f56"}
Sep 06 14:11:43.993809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:43.993747    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" podStartSLOduration=1.9937323 podStartE2EDuration="1.9937323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:43.991672919 +0000 UTC m=+841.017738096" watchObservedRunningTime="2026-09-06 14:11:43.9937323 +0000 UTC m=+841.019797474"
Sep 06 14:11:46.545855 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.545821    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"]
Sep 06 14:11:46.550244 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.550225    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:46.552309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.552289    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:11:46.552512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.552499    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:11:46.556004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.555979    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"]
Sep 06 14:11:46.646484 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.646450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n\" (UID: \"23db40ad-13f9-4142-8b62-f4ef2be6d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:46.746898 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.746858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n\" (UID: \"23db40ad-13f9-4142-8b62-f4ef2be6d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:46.747470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.747450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n\" (UID: \"23db40ad-13f9-4142-8b62-f4ef2be6d2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:46.862309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:46.862230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:47.133885 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:47.133794    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"]
Sep 06 14:11:47.136072 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:11:47.136038    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23db40ad_13f9_4142_8b62_f4ef2be6d2d9.slice/crio-10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6 WatchSource:0}: Error finding container 10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6: Status 404 returned error can't find the container with id 10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6
Sep 06 14:11:47.995463 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:47.995426    2576 generic.go:356] "Generic (PLEG): container finished" podID="23db40ad-13f9-4142-8b62-f4ef2be6d2d9" containerID="2c8821a23ac0a748a80d5295443175425b526755ea1944e6d216ce051eff3b59" exitCode=0
Sep 06 14:11:47.995873 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:47.995513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n" event={"ID":"23db40ad-13f9-4142-8b62-f4ef2be6d2d9","Type":"ContainerDied","Data":"2c8821a23ac0a748a80d5295443175425b526755ea1944e6d216ce051eff3b59"}
Sep 06 14:11:47.995873 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:47.995563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n" event={"ID":"23db40ad-13f9-4142-8b62-f4ef2be6d2d9","Type":"ContainerStarted","Data":"10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6"}
Sep 06 14:11:49.135098 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:49.135065    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:49.170981 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:49.170931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca\") pod \"23db40ad-13f9-4142-8b62-f4ef2be6d2d9\" (UID: \"23db40ad-13f9-4142-8b62-f4ef2be6d2d9\") "
Sep 06 14:11:49.171283 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:49.171261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23db40ad-13f9-4142-8b62-f4ef2be6d2d9" (UID: "23db40ad-13f9-4142-8b62-f4ef2be6d2d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:11:49.271924 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:49.271830    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23db40ad-13f9-4142-8b62-f4ef2be6d2d9-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:11:50.005859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:50.005829    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"
Sep 06 14:11:50.006038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:50.005859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n" event={"ID":"23db40ad-13f9-4142-8b62-f4ef2be6d2d9","Type":"ContainerDied","Data":"10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6"}
Sep 06 14:11:50.006038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:50.005911    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f9a35f99f43c06e377b0ef1a0714f93ee44302981254fbc719877920f6e8d6"
Sep 06 14:11:52.499509 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:52.499459    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"]
Sep 06 14:11:52.501992 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:52.501967    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c0e08d9a-5kj8n"]
Sep 06 14:11:53.106347 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.106303    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:53.106347 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.106357    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:53.118755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.118727    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:53.318016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.317981    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:11:53.318299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.318285    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23db40ad-13f9-4142-8b62-f4ef2be6d2d9" containerName="revoke-keys"
Sep 06 14:11:53.318299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.318299    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23db40ad-13f9-4142-8b62-f4ef2be6d2d9" containerName="revoke-keys"
Sep 06 14:11:53.318411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.318368    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23db40ad-13f9-4142-8b62-f4ef2be6d2d9" containerName="revoke-keys"
Sep 06 14:11:53.321416 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.321394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.323612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.323587    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d954782c-openshift-default-dockercfg-6pqwt\""
Sep 06 14:11:53.332921 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.332881    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:11:53.409049 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.408949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409049 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvkb\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.409590 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.409299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510115 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvkb\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510326    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.510572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.511016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.511016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.511016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510756    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.511016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.510880    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.511197 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.511069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.512661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.512638    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.512799 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.512780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.519230 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.519204    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.519547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.519522    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvkb\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb\") pod \"e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") " pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.610150 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.610115    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23db40ad-13f9-4142-8b62-f4ef2be6d2d9" path="/var/lib/kubelet/pods/23db40ad-13f9-4142-8b62-f4ef2be6d2d9/volumes"
Sep 06 14:11:53.633597 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.633564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:53.908837 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.908811    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:11:53.910706 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:11:53.910677    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345b6d95_8b19_47ed_a87b_dda315bd4045.slice/crio-de5af798ff5a1f5063746e05a248f630b8c48320376bd283d565c0e0b1510d3e WatchSource:0}: Error finding container de5af798ff5a1f5063746e05a248f630b8c48320376bd283d565c0e0b1510d3e: Status 404 returned error can't find the container with id de5af798ff5a1f5063746e05a248f630b8c48320376bd283d565c0e0b1510d3e
Sep 06 14:11:53.913018 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.912984    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:53.913088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.913056    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:53.913088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:53.913084    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:11:54.033153 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.033118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerStarted","Data":"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"}
Sep 06 14:11:54.033361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.033201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerStarted","Data":"de5af798ff5a1f5063746e05a248f630b8c48320376bd283d565c0e0b1510d3e"}
Sep 06 14:11:54.049453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.049424    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:11:54.053871 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.053817    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podStartSLOduration=1.053796248 podStartE2EDuration="1.053796248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:54.05092192 +0000 UTC m=+851.076987098" watchObservedRunningTime="2026-09-06 14:11:54.053796248 +0000 UTC m=+851.079861425"
Sep 06 14:11:54.634417 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.634383    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:54.639274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:54.639248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:55.037434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:55.037391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:55.038232 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:55.038211    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:11:56.479682 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.479642    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:11:56.483102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.483079    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.485391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.485367    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1482ae\"/\"kube-root-ca.crt\""
Sep 06 14:11:56.486298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.486276    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-1482ae\"/\"openshift-service-ca.crt\""
Sep 06 14:11:56.486374 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.486281    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1482ae\"/\"default-dockercfg-4wk69\""
Sep 06 14:11:56.486374 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.486281    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-1482ae\"/\"autores-model-1482ae-kserve-self-signed-certs\""
Sep 06 14:11:56.493283 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.493256    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:11:56.539961 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.539926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.539961 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.539967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.540167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.539985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.540167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.540010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.540167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.540116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lxk\" (UniqueName: \"kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.540167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.540154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641679 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641679 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641693    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lxk\" (UniqueName: \"kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.641914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.641813    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.642213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.642188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.642265 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.642229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.642265 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.642248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.644028 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.644006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.644197 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.644181    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.649756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.649726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lxk\" (UniqueName: \"kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk\") pod \"autores-model-1482ae-kserve-74fb5f767f-7vsmf\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") " pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:56.793898 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:56.793805    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:11:57.075818 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:57.075790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:11:57.078133 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:11:57.078104    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f33c316_8c66_4e2a_9d6a_44a19d7a0549.slice/crio-26a64de28458796100cd69545e0b19ada3b45a7be3704c7cf380e0baa112d0c5 WatchSource:0}: Error finding container 26a64de28458796100cd69545e0b19ada3b45a7be3704c7cf380e0baa112d0c5: Status 404 returned error can't find the container with id 26a64de28458796100cd69545e0b19ada3b45a7be3704c7cf380e0baa112d0c5
Sep 06 14:11:58.051148 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:58.051109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" event={"ID":"2f33c316-8c66-4e2a-9d6a-44a19d7a0549","Type":"ContainerStarted","Data":"cc759d5e7f38f13f28862ce2f449c068d56b5b2399b06e23fe2c9e3d4f11c89f"}
Sep 06 14:11:58.051505 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:58.051160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" event={"ID":"2f33c316-8c66-4e2a-9d6a-44a19d7a0549","Type":"ContainerStarted","Data":"26a64de28458796100cd69545e0b19ada3b45a7be3704c7cf380e0baa112d0c5"}
Sep 06 14:11:58.065906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:11:58.065823    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" podStartSLOduration=2.065803148 podStartE2EDuration="2.065803148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:11:58.065260825 +0000 UTC m=+855.091326011" watchObservedRunningTime="2026-09-06 14:11:58.065803148 +0000 UTC m=+855.091868323"
Sep 06 14:12:06.794939 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:06.794893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:06.795355 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:06.794969    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:06.807607 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:06.807579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:07.102679 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:07.102596    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:11.078967 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:11.078906    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:12:11.079449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:11.078985    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:12:12.118107 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.118070    2576 generic.go:356] "Generic (PLEG): container finished" podID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerID="b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5" exitCode=0
Sep 06 14:12:12.118531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.118142    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerDied","Data":"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"}
Sep 06 14:12:12.118588 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.118546    2576 scope.go:117] "RemoveContainer" containerID="b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"
Sep 06 14:12:12.125087 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.125052    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:12:12.125163 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.125131    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:12:12.125199 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:12.125168    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:12:13.128461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:13.127701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerStarted","Data":"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"}
Sep 06 14:12:13.634108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:13.634058    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:13.634525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:13.634140    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 06 14:12:13.634724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:13.634691    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 06 14:12:14.634845 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:14.634804    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 06 14:12:14.635462 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:14.635427    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 06 14:12:15.635124 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:15.635080    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 06 14:12:15.635718 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:15.635686    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 06 14:12:16.634696 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:16.634650    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 06 14:12:16.635061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:16.635019    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 06 14:12:17.634417 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:17.634362    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 06 14:12:17.634937 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:17.634473    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 06 14:12:18.637533 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:18.637500    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:18.637907 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:18.637788    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:18.638553 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:18.638532    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:20.986472 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.986435    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"]
Sep 06 14:12:20.989962 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.989940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:20.990445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.990414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d954782c-q7xtz\" (UID: \"a0a1755f-79fe-456c-8153-f0a47376a93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:20.992108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.992083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:12:20.992223 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.992157    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:12:20.995668 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:20.995643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"]
Sep 06 14:12:21.091877 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:21.091828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d954782c-q7xtz\" (UID: \"a0a1755f-79fe-456c-8153-f0a47376a93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:21.092456 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:21.092434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d954782c-q7xtz\" (UID: \"a0a1755f-79fe-456c-8153-f0a47376a93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:21.301731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:21.301636    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:21.572968 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:21.572939    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"]
Sep 06 14:12:21.574999 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:12:21.574954    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a1755f_79fe_456c_8153_f0a47376a93a.slice/crio-889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347 WatchSource:0}: Error finding container 889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347: Status 404 returned error can't find the container with id 889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347
Sep 06 14:12:22.176929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:22.176895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz" event={"ID":"a0a1755f-79fe-456c-8153-f0a47376a93a","Type":"ContainerStarted","Data":"c7d0f2859af2a784178617c3a225455d62a31e26016a3906c000dc306a1423a8"}
Sep 06 14:12:22.177380 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:22.176945    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz" event={"ID":"a0a1755f-79fe-456c-8153-f0a47376a93a","Type":"ContainerStarted","Data":"889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347"}
Sep 06 14:12:22.191720 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:22.191658    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz" podStartSLOduration=2.191640895 podStartE2EDuration="2.191640895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:22.189670341 +0000 UTC m=+879.215735527" watchObservedRunningTime="2026-09-06 14:12:22.191640895 +0000 UTC m=+879.217706071"
Sep 06 14:12:23.184186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:23.184147    2576 generic.go:356] "Generic (PLEG): container finished" podID="a0a1755f-79fe-456c-8153-f0a47376a93a" containerID="c7d0f2859af2a784178617c3a225455d62a31e26016a3906c000dc306a1423a8" exitCode=0
Sep 06 14:12:23.184186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:23.184183    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz" event={"ID":"a0a1755f-79fe-456c-8153-f0a47376a93a","Type":"ContainerDied","Data":"c7d0f2859af2a784178617c3a225455d62a31e26016a3906c000dc306a1423a8"}
Sep 06 14:12:24.314111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:24.314082    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:24.417321 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:24.417284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca\") pod \"a0a1755f-79fe-456c-8153-f0a47376a93a\" (UID: \"a0a1755f-79fe-456c-8153-f0a47376a93a\") "
Sep 06 14:12:24.417669 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:24.417643    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0a1755f-79fe-456c-8153-f0a47376a93a" (UID: "a0a1755f-79fe-456c-8153-f0a47376a93a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:12:24.518589 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:24.518496    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0a1755f-79fe-456c-8153-f0a47376a93a-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:25.193846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:25.193811    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"
Sep 06 14:12:25.193846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:25.193834    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz" event={"ID":"a0a1755f-79fe-456c-8153-f0a47376a93a","Type":"ContainerDied","Data":"889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347"}
Sep 06 14:12:25.194102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:25.193882    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="889466e6dac5b2ffc5572caa3c82534c00ebebe0ee8d5e53d6200c30c8e61347"
Sep 06 14:12:27.352675 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:27.352639    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"]
Sep 06 14:12:27.357289 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:27.357262    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d954782c-q7xtz"]
Sep 06 14:12:27.609574 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:27.609483    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a1755f-79fe-456c-8153-f0a47376a93a" path="/var/lib/kubelet/pods/a0a1755f-79fe-456c-8153-f0a47376a93a/volumes"
Sep 06 14:12:28.386064 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:28.386026    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:12:28.386475 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:28.386312    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" containerID="cri-o://953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4" gracePeriod=30
Sep 06 14:12:30.675777 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.675731    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:12:30.676133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.676065    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a1755f-79fe-456c-8153-f0a47376a93a" containerName="revoke-keys"
Sep 06 14:12:30.676133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.676078    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a1755f-79fe-456c-8153-f0a47376a93a" containerName="revoke-keys"
Sep 06 14:12:30.676206 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.676138    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a1755f-79fe-456c-8153-f0a47376a93a" containerName="revoke-keys"
Sep 06 14:12:30.680123 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.680104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.683038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.683012    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dfd6b\"/\"openshift-service-ca.crt\""
Sep 06 14:12:30.683038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.683028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dfd6b\"/\"default-dockercfg-98mjq\""
Sep 06 14:12:30.683038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.683038    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1dfd6b\"/\"kube-root-ca.crt\""
Sep 06 14:12:30.683262 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.683014    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1dfd6b\"/\"test-model-1dfd6b-kserve-self-signed-certs\""
Sep 06 14:12:30.689259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.689229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:12:30.774662 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.774622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.774867 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.774839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.774917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.774879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.774994 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.774978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.775076 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.775037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.775076 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.775070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5c2\" (UniqueName: \"kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876291 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5c2\" (UniqueName: \"kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876870 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876976 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.876976 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.876927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.878675 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.878651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.878848 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.878830    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:30.883288 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:30.883269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5c2\" (UniqueName: \"kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2\") pod \"test-model-1dfd6b-kserve-677799f7f5-s65st\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") " pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:31.010647 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:31.010555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:31.296359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:31.296330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:12:31.298324 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:12:31.298298    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9774d0ca_ebea_460b_ad51_270b331abb6c.slice/crio-44cc47afb7751ac325f981048bdd3f84a39207f505640de2bd510ec388a8a9af WatchSource:0}: Error finding container 44cc47afb7751ac325f981048bdd3f84a39207f505640de2bd510ec388a8a9af: Status 404 returned error can't find the container with id 44cc47afb7751ac325f981048bdd3f84a39207f505640de2bd510ec388a8a9af
Sep 06 14:12:32.228902 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:32.228859    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" event={"ID":"9774d0ca-ebea-460b-ad51-270b331abb6c","Type":"ContainerStarted","Data":"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"}
Sep 06 14:12:32.228902 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:32.228912    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" event={"ID":"9774d0ca-ebea-460b-ad51-270b331abb6c","Type":"ContainerStarted","Data":"44cc47afb7751ac325f981048bdd3f84a39207f505640de2bd510ec388a8a9af"}
Sep 06 14:12:32.246230 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:32.246174    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" podStartSLOduration=2.246159198 podStartE2EDuration="2.246159198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:12:32.245407843 +0000 UTC m=+889.271473018" watchObservedRunningTime="2026-09-06 14:12:32.246159198 +0000 UTC m=+889.272224373"
Sep 06 14:12:33.663423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.663395    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:33.703909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.703878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.703909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.703914    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.703938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.703962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704046    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704071    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704102    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704156    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704474 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfvkb\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb\") pod \"345b6d95-8b19-47ed-a87b-dda315bd4045\" (UID: \"345b6d95-8b19-47ed-a87b-dda315bd4045\") "
Sep 06 14:12:33.704474 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704214    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data" (OuterVolumeSpecName: "istio-data") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:33.704474 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704325    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:33.704474 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:33.704680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704481    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:12:33.704680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704601    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/345b6d95-8b19-47ed-a87b-dda315bd4045-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.704680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704621    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.704680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704634    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.704680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704649    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.704967 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.704833    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:33.706655 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.706630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:12:33.706779 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.706730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token" (OuterVolumeSpecName: "istio-token") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:12:33.706859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.706841    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb" (OuterVolumeSpecName: "kube-api-access-hfvkb") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "kube-api-access-hfvkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:12:33.706919 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.706840    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "345b6d95-8b19-47ed-a87b-dda315bd4045" (UID: "345b6d95-8b19-47ed-a87b-dda315bd4045"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:33.805226 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.805188    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfvkb\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-kube-api-access-hfvkb\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.805226 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.805221    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.805226 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.805232    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.805446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.805239    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:33.805446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:33.805248    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/345b6d95-8b19-47ed-a87b-dda315bd4045-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:34.073149 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.073110    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:12:34.239819 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.239783    2576 generic.go:356] "Generic (PLEG): container finished" podID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerID="953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4" exitCode=0
Sep 06 14:12:34.239985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.239822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerDied","Data":"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"}
Sep 06 14:12:34.239985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.239835    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"
Sep 06 14:12:34.239985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.239877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" event={"ID":"345b6d95-8b19-47ed-a87b-dda315bd4045","Type":"ContainerDied","Data":"de5af798ff5a1f5063746e05a248f630b8c48320376bd283d565c0e0b1510d3e"}
Sep 06 14:12:34.239985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.239904    2576 scope.go:117] "RemoveContainer" containerID="953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"
Sep 06 14:12:34.249002 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.248984    2576 scope.go:117] "RemoveContainer" containerID="b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"
Sep 06 14:12:34.257207 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.257187    2576 scope.go:117] "RemoveContainer" containerID="953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"
Sep 06 14:12:34.257496 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:12:34.257476    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4\": container with ID starting with 953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4 not found: ID does not exist" containerID="953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"
Sep 06 14:12:34.257543 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.257506    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4"} err="failed to get container status \"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4\": rpc error: code = NotFound desc = could not find container \"953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4\": container with ID starting with 953b6f6099555fdfd3da16cb99f2e13da0ed6e0b1d5a60cd03bf0c14df12d7a4 not found: ID does not exist"
Sep 06 14:12:34.257543 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.257523    2576 scope.go:117] "RemoveContainer" containerID="b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"
Sep 06 14:12:34.257809 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:12:34.257779    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5\": container with ID starting with b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5 not found: ID does not exist" containerID="b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"
Sep 06 14:12:34.257862 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.257811    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5"} err="failed to get container status \"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5\": rpc error: code = NotFound desc = could not find container \"b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5\": container with ID starting with b520a403290e711c32e972a97d8b4f501a600e16328f1ded9ac6a980813f53d5 not found: ID does not exist"
Sep 06 14:12:34.265144 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.265118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:12:34.268600 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.268577    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx"]
Sep 06 14:12:34.639973 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.639914    2576 patch_prober.go:28] interesting pod/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:12:34.640162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:34.640002    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d954782c-openshift-default-64b5f96744-gkgqx" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:12:35.610396 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:35.610364    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" path="/var/lib/kubelet/pods/345b6d95-8b19-47ed-a87b-dda315bd4045/volumes"
Sep 06 14:12:37.871267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:37.871229    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:12:40.180036 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:40.180000    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:12:41.011375 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:41.011339    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:41.011585 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:41.011400    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:41.024294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:41.024263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:41.289798 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:41.289695    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:12:43.454666 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:43.454636    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:12:43.597319 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:43.597293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:12:49.093315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.093245    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" podUID="7fbf8011-000b-48aa-a677-50f02bfe041f" containerName="main" containerID="cri-o://0eb492f21aa68a00670c9c27120b06390abed5eb2115d52aff720643244322e8" gracePeriod=45
Sep 06 14:12:49.316414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.316381    2576 generic.go:356] "Generic (PLEG): container finished" podID="7fbf8011-000b-48aa-a677-50f02bfe041f" containerID="0eb492f21aa68a00670c9c27120b06390abed5eb2115d52aff720643244322e8" exitCode=0
Sep 06 14:12:49.316575 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.316422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" event={"ID":"7fbf8011-000b-48aa-a677-50f02bfe041f","Type":"ContainerDied","Data":"0eb492f21aa68a00670c9c27120b06390abed5eb2115d52aff720643244322e8"}
Sep 06 14:12:49.344505 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.344439    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:12:49.456971 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.456923    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457008    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457036    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457114    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrmxw\" (UniqueName: \"kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs\") pod \"7fbf8011-000b-48aa-a677-50f02bfe041f\" (UID: \"7fbf8011-000b-48aa-a677-50f02bfe041f\") "
Sep 06 14:12:49.457370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:49.457370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:49.457517 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home" (OuterVolumeSpecName: "home") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:49.457622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457604    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:49.457671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457626    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:49.457671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.457638    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:49.459246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.459222    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm" (OuterVolumeSpecName: "dshm") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:49.459360 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.459338    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:12:49.459401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.459339    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw" (OuterVolumeSpecName: "kube-api-access-wrmxw") pod "7fbf8011-000b-48aa-a677-50f02bfe041f" (UID: "7fbf8011-000b-48aa-a677-50f02bfe041f"). InnerVolumeSpecName "kube-api-access-wrmxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:12:49.558756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.558712    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrmxw\" (UniqueName: \"kubernetes.io/projected/7fbf8011-000b-48aa-a677-50f02bfe041f-kube-api-access-wrmxw\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:49.558756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.558747    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fbf8011-000b-48aa-a677-50f02bfe041f-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:49.558756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:49.558778    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fbf8011-000b-48aa-a677-50f02bfe041f-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:50.322780 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:50.322725    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"
Sep 06 14:12:50.322780 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:50.322725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7" event={"ID":"7fbf8011-000b-48aa-a677-50f02bfe041f","Type":"ContainerDied","Data":"c9087cbec3601c3a0ce0dacd28e75220c6c0c5d43241bdce2e92473018a3bf08"}
Sep 06 14:12:50.323258 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:50.322815    2576 scope.go:117] "RemoveContainer" containerID="0eb492f21aa68a00670c9c27120b06390abed5eb2115d52aff720643244322e8"
Sep 06 14:12:50.336820 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:50.336791    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:12:50.340468 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:50.340440    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-8318d32e-kserve-cb57d48c7-xb9s7"]
Sep 06 14:12:51.609419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:51.609386    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbf8011-000b-48aa-a677-50f02bfe041f" path="/var/lib/kubelet/pods/7fbf8011-000b-48aa-a677-50f02bfe041f/volumes"
Sep 06 14:12:52.890741 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:52.890706    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" podUID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" containerName="main" containerID="cri-o://2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827" gracePeriod=45
Sep 06 14:12:53.130895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.130866    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:12:53.295994 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.295961    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296090    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz4rz\" (UniqueName: \"kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296153    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296368 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296368 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home" (OuterVolumeSpecName: "home") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:53.296476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296408    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache\") pod \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\" (UID: \"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33\") "
Sep 06 14:12:53.296476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296453    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:53.296823 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296788    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:53.296823 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296807    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.296972 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.296830    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.298408 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.298385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm" (OuterVolumeSpecName: "dshm") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:53.298479 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.298408    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:12:53.298521 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.298482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz" (OuterVolumeSpecName: "kube-api-access-gz4rz") pod "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" (UID: "1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33"). InnerVolumeSpecName "kube-api-access-gz4rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:12:53.339498 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.339465    2576 generic.go:356] "Generic (PLEG): container finished" podID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" containerID="2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827" exitCode=0
Sep 06 14:12:53.339667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.339504    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"
Sep 06 14:12:53.339667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.339523    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" event={"ID":"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33","Type":"ContainerDied","Data":"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"}
Sep 06 14:12:53.339667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.339560    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4" event={"ID":"1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33","Type":"ContainerDied","Data":"f105eb35a97f4a35ec1b0786662cef95284fdd16a4591c6f5411af4abb072f56"}
Sep 06 14:12:53.339667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.339578    2576 scope.go:117] "RemoveContainer" containerID="2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"
Sep 06 14:12:53.348566 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.348548    2576 scope.go:117] "RemoveContainer" containerID="2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"
Sep 06 14:12:53.348907 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:12:53.348887    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827\": container with ID starting with 2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827 not found: ID does not exist" containerID="2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"
Sep 06 14:12:53.348958 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.348916    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827"} err="failed to get container status \"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827\": rpc error: code = NotFound desc = could not find container \"2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827\": container with ID starting with 2412363b15a75dd7a3ebd8af2c61ea6149dd04562c869ff914c9349eed826827 not found: ID does not exist"
Sep 06 14:12:53.358868 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.358834    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:12:53.362673 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.362647    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-8318d32e-kserve-bbdd64b9f-kqsz4"]
Sep 06 14:12:53.397947 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.397909    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.397947 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.397942    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4rz\" (UniqueName: \"kubernetes.io/projected/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-kube-api-access-gz4rz\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.397947 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.397951    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.398164 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.397962    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:53.610932 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:53.610847    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" path="/var/lib/kubelet/pods/1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33/volumes"
Sep 06 14:12:55.199753 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.199691    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" podUID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" containerName="main" containerID="cri-o://cc759d5e7f38f13f28862ce2f449c068d56b5b2399b06e23fe2c9e3d4f11c89f" gracePeriod=45
Sep 06 14:12:55.352434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.352402    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" containerID="cc759d5e7f38f13f28862ce2f449c068d56b5b2399b06e23fe2c9e3d4f11c89f" exitCode=0
Sep 06 14:12:55.352597 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.352481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" event={"ID":"2f33c316-8c66-4e2a-9d6a-44a19d7a0549","Type":"ContainerDied","Data":"cc759d5e7f38f13f28862ce2f449c068d56b5b2399b06e23fe2c9e3d4f11c89f"}
Sep 06 14:12:55.454480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.454419    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:55.621291 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621265    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621349    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6lxk\" (UniqueName: \"kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621404    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621434    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home" (OuterVolumeSpecName: "home") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:55.621652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621547    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir\") pod \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\" (UID: \"2f33c316-8c66-4e2a-9d6a-44a19d7a0549\") "
Sep 06 14:12:55.621752 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621731    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:55.621827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621738    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:55.621974 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621954    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:55.622039 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621977    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:55.622039 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.621990    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:55.623619 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.623594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk" (OuterVolumeSpecName: "kube-api-access-g6lxk") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "kube-api-access-g6lxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:12:55.623891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.623876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm" (OuterVolumeSpecName: "dshm") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:12:55.623964 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.623899    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f33c316-8c66-4e2a-9d6a-44a19d7a0549" (UID: "2f33c316-8c66-4e2a-9d6a-44a19d7a0549"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:12:55.722581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.722479    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:55.722581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.722516    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6lxk\" (UniqueName: \"kubernetes.io/projected/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-kube-api-access-g6lxk\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:55.722581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:55.722531    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f33c316-8c66-4e2a-9d6a-44a19d7a0549-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:12:56.358220 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.358188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf" event={"ID":"2f33c316-8c66-4e2a-9d6a-44a19d7a0549","Type":"ContainerDied","Data":"26a64de28458796100cd69545e0b19ada3b45a7be3704c7cf380e0baa112d0c5"}
Sep 06 14:12:56.358636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.358233    2576 scope.go:117] "RemoveContainer" containerID="cc759d5e7f38f13f28862ce2f449c068d56b5b2399b06e23fe2c9e3d4f11c89f"
Sep 06 14:12:56.358636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.358200    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"
Sep 06 14:12:56.377988 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.377954    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:12:56.381090 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.381067    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-1482ae/autores-model-1482ae-kserve-74fb5f767f-7vsmf"]
Sep 06 14:12:56.850233 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850199    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"]
Sep 06 14:12:56.850514 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850502    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850516    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850525    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850531    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850538    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbf8011-000b-48aa-a677-50f02bfe041f" containerName="main"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850544    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbf8011-000b-48aa-a677-50f02bfe041f" containerName="main"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850549    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" containerName="main"
Sep 06 14:12:56.850559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850555    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" containerName="main"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850564    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" containerName="main"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850570    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" containerName="main"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850628    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850635    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" containerName="main"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850643    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="345b6d95-8b19-47ed-a87b-dda315bd4045" containerName="istio-proxy"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850649    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbf8011-000b-48aa-a677-50f02bfe041f" containerName="main"
Sep 06 14:12:56.850859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.850655    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bcb6cf9-e1f4-43ef-87b5-490a8d5d2f33" containerName="main"
Sep 06 14:12:56.854687 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.854664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:56.856598 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.856579    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:12:56.856804 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.856784    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:12:56.859873 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:56.859845    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"]
Sep 06 14:12:57.036511 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.036472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46\" (UID: \"80aeba7b-e830-434d-8f56-f151153980f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:57.137690 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.137593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46\" (UID: \"80aeba7b-e830-434d-8f56-f151153980f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:57.138304 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.138284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46\" (UID: \"80aeba7b-e830-434d-8f56-f151153980f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:57.166629 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.166594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:57.436351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.436258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"]
Sep 06 14:12:57.438779 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:12:57.438738    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80aeba7b_e830_434d_8f56_f151153980f3.slice/crio-59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d WatchSource:0}: Error finding container 59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d: Status 404 returned error can't find the container with id 59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d
Sep 06 14:12:57.609926 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:57.609894    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f33c316-8c66-4e2a-9d6a-44a19d7a0549" path="/var/lib/kubelet/pods/2f33c316-8c66-4e2a-9d6a-44a19d7a0549/volumes"
Sep 06 14:12:58.371614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:58.371517    2576 generic.go:356] "Generic (PLEG): container finished" podID="80aeba7b-e830-434d-8f56-f151153980f3" containerID="a8d01a73816e69c2ea0237b88716436367f74226f9385c0eea27b9493a424acf" exitCode=0
Sep 06 14:12:58.371781 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:58.371604    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46" event={"ID":"80aeba7b-e830-434d-8f56-f151153980f3","Type":"ContainerDied","Data":"a8d01a73816e69c2ea0237b88716436367f74226f9385c0eea27b9493a424acf"}
Sep 06 14:12:58.371781 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:58.371685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46" event={"ID":"80aeba7b-e830-434d-8f56-f151153980f3","Type":"ContainerStarted","Data":"59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d"}
Sep 06 14:12:59.501749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:59.501719    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:12:59.661821 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:59.661708    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca\") pod \"80aeba7b-e830-434d-8f56-f151153980f3\" (UID: \"80aeba7b-e830-434d-8f56-f151153980f3\") "
Sep 06 14:12:59.662112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:59.662086    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80aeba7b-e830-434d-8f56-f151153980f3" (UID: "80aeba7b-e830-434d-8f56-f151153980f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:12:59.763380 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:12:59.763335    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80aeba7b-e830-434d-8f56-f151153980f3-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:00.383473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:00.383440    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"
Sep 06 14:13:00.383473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:00.383467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46" event={"ID":"80aeba7b-e830-434d-8f56-f151153980f3","Type":"ContainerDied","Data":"59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d"}
Sep 06 14:13:00.383689 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:00.383525    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e075605dfaac541973253c6f6a1e9f12e37946c92b55495061569da31ed67d"
Sep 06 14:13:03.904690 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:03.904649    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"]
Sep 06 14:13:03.907362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:03.907333    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b1ed35f1-95x46"]
Sep 06 14:13:05.609458 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:05.609426    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80aeba7b-e830-434d-8f56-f151153980f3" path="/var/lib/kubelet/pods/80aeba7b-e830-434d-8f56-f151153980f3/volumes"
Sep 06 14:13:10.687054 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.687012    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:13:10.687494 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.687317    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80aeba7b-e830-434d-8f56-f151153980f3" containerName="revoke-keys"
Sep 06 14:13:10.687494 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.687328    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80aeba7b-e830-434d-8f56-f151153980f3" containerName="revoke-keys"
Sep 06 14:13:10.687494 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.687382    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80aeba7b-e830-434d-8f56-f151153980f3" containerName="revoke-keys"
Sep 06 14:13:10.690239 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.690216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.693646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.693619    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ec585\"/\"test-model-5ec585-kserve-self-signed-certs\""
Sep 06 14:13:10.693794 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.693657    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ec585\"/\"openshift-service-ca.crt\""
Sep 06 14:13:10.693794 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.693699    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5ec585\"/\"kube-root-ca.crt\""
Sep 06 14:13:10.693974 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.693954    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5ec585\"/\"default-dockercfg-9pxnq\""
Sep 06 14:13:10.698425 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.698395    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:13:10.761774 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.761956 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.761956 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.761956 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.761956 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761932    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nm6\" (UniqueName: \"kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.762091 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.761984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863292 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863488 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863488 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nm6\" (UniqueName: \"kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863822 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.863982 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.863935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.865926 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.865896    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.866026 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.865994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:10.870583 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:10.870557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nm6\" (UniqueName: \"kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6\") pod \"test-model-5ec585-kserve-84c5b7cd59-mprkm\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") " pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:11.000501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:11.000400    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:11.273705 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:11.273678    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:13:11.275811 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:13:11.275783    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbfcd43f_1c39_486a_b63b_92ad6e3b3934.slice/crio-b6b808f9ad3d374771d6417014d2e45389fad1d721dfe49952ba6705f81fe466 WatchSource:0}: Error finding container b6b808f9ad3d374771d6417014d2e45389fad1d721dfe49952ba6705f81fe466: Status 404 returned error can't find the container with id b6b808f9ad3d374771d6417014d2e45389fad1d721dfe49952ba6705f81fe466
Sep 06 14:13:11.440130 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:11.440092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" event={"ID":"cbfcd43f-1c39-486a-b63b-92ad6e3b3934","Type":"ContainerStarted","Data":"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"}
Sep 06 14:13:11.440130 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:11.440136    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" event={"ID":"cbfcd43f-1c39-486a-b63b-92ad6e3b3934","Type":"ContainerStarted","Data":"b6b808f9ad3d374771d6417014d2e45389fad1d721dfe49952ba6705f81fe466"}
Sep 06 14:13:11.455551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:11.455501    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" podStartSLOduration=1.455483299 podStartE2EDuration="1.455483299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:13:11.454623813 +0000 UTC m=+928.480688983" watchObservedRunningTime="2026-09-06 14:13:11.455483299 +0000 UTC m=+928.481548546"
Sep 06 14:13:21.000890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:21.000848    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:21.001324 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:21.001140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:21.013599 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:21.013566    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:21.497269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:21.497240    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:13:24.392319 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:24.392286    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:13:24.392746 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:24.392510    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy" containerID="cri-o://04b8076fdeafe534480b7b28bedd6045c07d03ca2415a955508c09baeff58801" gracePeriod=30
Sep 06 14:13:29.528665 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.528625    2576 generic.go:356] "Generic (PLEG): container finished" podID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerID="04b8076fdeafe534480b7b28bedd6045c07d03ca2415a955508c09baeff58801" exitCode=0
Sep 06 14:13:29.529140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.528690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerDied","Data":"04b8076fdeafe534480b7b28bedd6045c07d03ca2415a955508c09baeff58801"}
Sep 06 14:13:29.529140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.528755    2576 scope.go:117] "RemoveContainer" containerID="54d5bb4dda45b0e84d3b0d1ecbee1a8988bc68ddb81dd1431b226f35c44c9531"
Sep 06 14:13:29.642257 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.642214    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:13:29.753411 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753354    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753599 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753434    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753599 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753499    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753721 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753698    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knhnl\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753783    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753879 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753823    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753879 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753868    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753901    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.753986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.753953    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket\") pod \"cde04245-e65d-41b6-9dbb-4cbea2701f40\" (UID: \"cde04245-e65d-41b6-9dbb-4cbea2701f40\") "
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754291    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data" (OuterVolumeSpecName: "istio-data") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754299    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754383    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754430    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754450    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.754555 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754465    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.755053 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.754578    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:13:29.755746 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.755715    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:13:29.755969 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.755953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl" (OuterVolumeSpecName: "kube-api-access-knhnl") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "kube-api-access-knhnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:13:29.756045 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.756030    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token" (OuterVolumeSpecName: "istio-token") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:13:29.756159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.756138    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cde04245-e65d-41b6-9dbb-4cbea2701f40" (UID: "cde04245-e65d-41b6-9dbb-4cbea2701f40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:13:29.855595 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855559    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde04245-e65d-41b6-9dbb-4cbea2701f40-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.855595 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855591    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.855595 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855602    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.855878 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855610    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.855878 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855618    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde04245-e65d-41b6-9dbb-4cbea2701f40-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:29.855878 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:29.855626    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knhnl\" (UniqueName: \"kubernetes.io/projected/cde04245-e65d-41b6-9dbb-4cbea2701f40-kube-api-access-knhnl\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:13:30.534985 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:30.534960    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"
Sep 06 14:13:30.535397 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:30.534958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg" event={"ID":"cde04245-e65d-41b6-9dbb-4cbea2701f40","Type":"ContainerDied","Data":"50d08b7a7a33dfcd787b6d3eb9c6f21d19d38e357c2892af95de6287d3d95424"}
Sep 06 14:13:30.535397 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:30.535087    2576 scope.go:117] "RemoveContainer" containerID="04b8076fdeafe534480b7b28bedd6045c07d03ca2415a955508c09baeff58801"
Sep 06 14:13:30.555594 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:30.555563    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:13:30.565841 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:30.564027    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-1482ae-openshift-default-7d4d7d7cdd-9d4pg"]
Sep 06 14:13:31.610331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:13:31.610298    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" path="/var/lib/kubelet/pods/cde04245-e65d-41b6-9dbb-4cbea2701f40/volumes"
Sep 06 14:14:00.192151 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.192109    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:00.192615 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.192559    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:00.192615 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.192574    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:00.192687 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.192667    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:00.197126 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.197097    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.200601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.200333    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-dfca8b\"/\"kube-root-ca.crt\""
Sep 06 14:14:00.200601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.200402    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-dfca8b\"/\"openshift-service-ca.crt\""
Sep 06 14:14:00.200601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.200411    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-dfca8b\"/\"explicit-model-dfca8b-kserve-self-signed-certs\""
Sep 06 14:14:00.200601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.200336    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-dfca8b\"/\"default-dockercfg-4l7fp\""
Sep 06 14:14:00.203016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.202991    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:00.324817 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.324744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.325015 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.324839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.325015 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.324877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.325015 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.324904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgh6b\" (UniqueName: \"kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.325015 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.324981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.325162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.325016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426013 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.425968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426013 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426261 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426261 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426261 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426261 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgh6b\" (UniqueName: \"kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426471 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.426538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.426524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.428395 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.428366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.428514 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.428461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.433058 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.433040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgh6b\" (UniqueName: \"kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b\") pod \"explicit-model-dfca8b-kserve-599787c985-47qr2\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") " pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.508222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.508122    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:00.780652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:00.780609    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:00.783118 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:00.783089    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98649653_bbca_4feb_8e3c_8082863ded50.slice/crio-2849aae94919e80679fee9932e12a5530917592f281a9c59f33956616893e58e WatchSource:0}: Error finding container 2849aae94919e80679fee9932e12a5530917592f281a9c59f33956616893e58e: Status 404 returned error can't find the container with id 2849aae94919e80679fee9932e12a5530917592f281a9c59f33956616893e58e
Sep 06 14:14:01.672423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:01.672379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" event={"ID":"98649653-bbca-4feb-8e3c-8082863ded50","Type":"ContainerStarted","Data":"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"}
Sep 06 14:14:01.672845 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:01.672433    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" event={"ID":"98649653-bbca-4feb-8e3c-8082863ded50","Type":"ContainerStarted","Data":"2849aae94919e80679fee9932e12a5530917592f281a9c59f33956616893e58e"}
Sep 06 14:14:01.687315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:01.687239    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" podStartSLOduration=1.687223025 podStartE2EDuration="1.687223025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:01.686312743 +0000 UTC m=+978.712377929" watchObservedRunningTime="2026-09-06 14:14:01.687223025 +0000 UTC m=+978.713288201"
Sep 06 14:14:06.191082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.190994    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:06.315402 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.315366    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"]
Sep 06 14:14:06.315703 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.315691    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:06.315747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.315705    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:06.315813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.315780    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde04245-e65d-41b6-9dbb-4cbea2701f40" containerName="istio-proxy"
Sep 06 14:14:06.318715 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.318698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:06.320970 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.320947    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:14:06.321067 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.320979    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:14:06.325181 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.325151    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"]
Sep 06 14:14:06.388653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.388611    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb\" (UID: \"f7d20336-4fc9-4cf9-b901-08bd1730e327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:06.490169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.490081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb\" (UID: \"f7d20336-4fc9-4cf9-b901-08bd1730e327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:06.490698 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.490676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb\" (UID: \"f7d20336-4fc9-4cf9-b901-08bd1730e327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:06.631114 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.631071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:06.920106 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:06.920082    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"]
Sep 06 14:14:06.921878 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:06.921849    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d20336_4fc9_4cf9_b901_08bd1730e327.slice/crio-f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8 WatchSource:0}: Error finding container f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8: Status 404 returned error can't find the container with id f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8
Sep 06 14:14:07.704336 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:07.704300    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7d20336-4fc9-4cf9-b901-08bd1730e327" containerID="848ee019ecd274a97350c3bf7e4057b8464a7582da640cf572e49d6d6f2a49d6" exitCode=0
Sep 06 14:14:07.704714 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:07.704347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb" event={"ID":"f7d20336-4fc9-4cf9-b901-08bd1730e327","Type":"ContainerDied","Data":"848ee019ecd274a97350c3bf7e4057b8464a7582da640cf572e49d6d6f2a49d6"}
Sep 06 14:14:07.704714 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:07.704390    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb" event={"ID":"f7d20336-4fc9-4cf9-b901-08bd1730e327","Type":"ContainerStarted","Data":"f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8"}
Sep 06 14:14:08.833643 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:08.833605    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:08.913073 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:08.913034    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca\") pod \"f7d20336-4fc9-4cf9-b901-08bd1730e327\" (UID: \"f7d20336-4fc9-4cf9-b901-08bd1730e327\") "
Sep 06 14:14:08.913410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:08.913387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7d20336-4fc9-4cf9-b901-08bd1730e327" (UID: "f7d20336-4fc9-4cf9-b901-08bd1730e327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:14:09.014407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:09.014327    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d20336-4fc9-4cf9-b901-08bd1730e327-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:09.715503 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:09.715465    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"
Sep 06 14:14:09.715503 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:09.715490    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb" event={"ID":"f7d20336-4fc9-4cf9-b901-08bd1730e327","Type":"ContainerDied","Data":"f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8"}
Sep 06 14:14:09.715720 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:09.715536    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f605a49c7b70430da75699068c9128224049b3ca03866914bf046ab7bd89d6b8"
Sep 06 14:14:10.508562 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:10.508513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:16.877908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:16.877868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"]
Sep 06 14:14:16.880461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:16.880434    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dfca8b-z7qvb"]
Sep 06 14:14:17.610252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:17.610220    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d20336-4fc9-4cf9-b901-08bd1730e327" path="/var/lib/kubelet/pods/f7d20336-4fc9-4cf9-b901-08bd1730e327/volumes"
Sep 06 14:14:18.796362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:18.796324    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:14:19.899347 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:19.899313    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:14:21.210567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.210507    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" podUID="98649653-bbca-4feb-8e3c-8082863ded50" containerName="main" containerID="cri-o://f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e" gracePeriod=45
Sep 06 14:14:21.461293 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.461229    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:21.525059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525073    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgh6b\" (UniqueName: \"kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525110    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525153    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525178    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir\") pod \"98649653-bbca-4feb-8e3c-8082863ded50\" (UID: \"98649653-bbca-4feb-8e3c-8082863ded50\") "
Sep 06 14:14:21.525515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525321    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache" (OuterVolumeSpecName: "model-cache") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:21.525515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525345    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home" (OuterVolumeSpecName: "home") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:21.525515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525442    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.525515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525457    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.525680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.525523    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:21.527447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.527416    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm" (OuterVolumeSpecName: "dshm") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:21.527447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.527431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b" (OuterVolumeSpecName: "kube-api-access-xgh6b") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "kube-api-access-xgh6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:21.527583 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.527455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98649653-bbca-4feb-8e3c-8082863ded50" (UID: "98649653-bbca-4feb-8e3c-8082863ded50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:14:21.626557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.626521    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98649653-bbca-4feb-8e3c-8082863ded50-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.626557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.626555    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.626786 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.626568    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98649653-bbca-4feb-8e3c-8082863ded50-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.626786 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.626581    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgh6b\" (UniqueName: \"kubernetes.io/projected/98649653-bbca-4feb-8e3c-8082863ded50-kube-api-access-xgh6b\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:21.774468 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.774381    2576 generic.go:356] "Generic (PLEG): container finished" podID="98649653-bbca-4feb-8e3c-8082863ded50" containerID="f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e" exitCode=0
Sep 06 14:14:21.774468 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.774430    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"
Sep 06 14:14:21.774652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.774467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" event={"ID":"98649653-bbca-4feb-8e3c-8082863ded50","Type":"ContainerDied","Data":"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"}
Sep 06 14:14:21.774652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.774513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2" event={"ID":"98649653-bbca-4feb-8e3c-8082863ded50","Type":"ContainerDied","Data":"2849aae94919e80679fee9932e12a5530917592f281a9c59f33956616893e58e"}
Sep 06 14:14:21.774652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.774532    2576 scope.go:117] "RemoveContainer" containerID="f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"
Sep 06 14:14:21.785100 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.784906    2576 scope.go:117] "RemoveContainer" containerID="f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"
Sep 06 14:14:21.785533 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:14:21.785500    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e\": container with ID starting with f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e not found: ID does not exist" containerID="f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"
Sep 06 14:14:21.785655 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.785545    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e"} err="failed to get container status \"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e\": rpc error: code = NotFound desc = could not find container \"f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e\": container with ID starting with f4b397917229e827b90586a8a94890644e1c9e3347b2a9da08ddb323fdf1957e not found: ID does not exist"
Sep 06 14:14:21.790140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.790118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:21.795616 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:21.795591    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-dfca8b/explicit-model-dfca8b-kserve-599787c985-47qr2"]
Sep 06 14:14:23.610221 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:23.610184    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98649653-bbca-4feb-8e3c-8082863ded50" path="/var/lib/kubelet/pods/98649653-bbca-4feb-8e3c-8082863ded50/volumes"
Sep 06 14:14:33.816344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:33.816296    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" podUID="9774d0ca-ebea-460b-ad51-270b331abb6c" containerName="main" containerID="cri-o://df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd" gracePeriod=45
Sep 06 14:14:34.070412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.070336    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:14:34.143449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143415    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j5c2\" (UniqueName: \"kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143490    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143513    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143535    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143585    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs\") pod \"9774d0ca-ebea-460b-ad51-270b331abb6c\" (UID: \"9774d0ca-ebea-460b-ad51-270b331abb6c\") "
Sep 06 14:14:34.143936 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:34.143936 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143830    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home" (OuterVolumeSpecName: "home") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:34.143936 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.143864    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:34.145732 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.145701    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:14:34.145839 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.145714    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm" (OuterVolumeSpecName: "dshm") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:34.145839 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.145780    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2" (OuterVolumeSpecName: "kube-api-access-7j5c2") pod "9774d0ca-ebea-460b-ad51-270b331abb6c" (UID: "9774d0ca-ebea-460b-ad51-270b331abb6c"). InnerVolumeSpecName "kube-api-access-7j5c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:34.244847 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244808    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.244847 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244839    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.244847 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244848    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9774d0ca-ebea-460b-ad51-270b331abb6c-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.244847 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244857    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5c2\" (UniqueName: \"kubernetes.io/projected/9774d0ca-ebea-460b-ad51-270b331abb6c-kube-api-access-7j5c2\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.245120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244865    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.245120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.244874    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9774d0ca-ebea-460b-ad51-270b331abb6c-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:34.833864 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.833823    2576 generic.go:356] "Generic (PLEG): container finished" podID="9774d0ca-ebea-460b-ad51-270b331abb6c" containerID="df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd" exitCode=0
Sep 06 14:14:34.834270 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.833886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" event={"ID":"9774d0ca-ebea-460b-ad51-270b331abb6c","Type":"ContainerDied","Data":"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"}
Sep 06 14:14:34.834270 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.833921    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st" event={"ID":"9774d0ca-ebea-460b-ad51-270b331abb6c","Type":"ContainerDied","Data":"44cc47afb7751ac325f981048bdd3f84a39207f505640de2bd510ec388a8a9af"}
Sep 06 14:14:34.834270 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.833937    2576 scope.go:117] "RemoveContainer" containerID="df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"
Sep 06 14:14:34.834270 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.833893    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"
Sep 06 14:14:34.842963 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.842932    2576 scope.go:117] "RemoveContainer" containerID="df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"
Sep 06 14:14:34.843259 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:14:34.843237    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd\": container with ID starting with df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd not found: ID does not exist" containerID="df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"
Sep 06 14:14:34.843308 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.843269    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd"} err="failed to get container status \"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd\": rpc error: code = NotFound desc = could not find container \"df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd\": container with ID starting with df913a014417cc71a6fc1d9a812b199bb9713a74f6fa3bfc8d5a359a9af12bfd not found: ID does not exist"
Sep 06 14:14:34.852826 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.852797    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:14:34.856726 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.856702    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1dfd6b/test-model-1dfd6b-kserve-677799f7f5-s65st"]
Sep 06 14:14:34.919454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:34.919396    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" podUID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" containerName="main" containerID="cri-o://4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c" gracePeriod=45
Sep 06 14:14:35.174397 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.174373    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:14:35.254281 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254281 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254300    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4nm6\" (UniqueName: \"kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254420    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs\") pod \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\" (UID: \"cbfcd43f-1c39-486a-b63b-92ad6e3b3934\") "
Sep 06 14:14:35.254731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:35.254731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home" (OuterVolumeSpecName: "home") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:35.254731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:35.254909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254737    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.254909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254751    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.254909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.254802    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.256616 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.256587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6" (OuterVolumeSpecName: "kube-api-access-s4nm6") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "kube-api-access-s4nm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:35.256616 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.256601    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:14:35.256793 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.256596    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm" (OuterVolumeSpecName: "dshm") pod "cbfcd43f-1c39-486a-b63b-92ad6e3b3934" (UID: "cbfcd43f-1c39-486a-b63b-92ad6e3b3934"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:35.355501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.355410    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nm6\" (UniqueName: \"kubernetes.io/projected/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-kube-api-access-s4nm6\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.355501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.355449    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.355501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.355461    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfcd43f-1c39-486a-b63b-92ad6e3b3934-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:35.610135 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.610060    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9774d0ca-ebea-460b-ad51-270b331abb6c" path="/var/lib/kubelet/pods/9774d0ca-ebea-460b-ad51-270b331abb6c/volumes"
Sep 06 14:14:35.839692 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.839653    2576 generic.go:356] "Generic (PLEG): container finished" podID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" containerID="4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c" exitCode=0
Sep 06 14:14:35.840186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.839702    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"
Sep 06 14:14:35.840186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.839744    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" event={"ID":"cbfcd43f-1c39-486a-b63b-92ad6e3b3934","Type":"ContainerDied","Data":"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"}
Sep 06 14:14:35.840186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.839816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm" event={"ID":"cbfcd43f-1c39-486a-b63b-92ad6e3b3934","Type":"ContainerDied","Data":"b6b808f9ad3d374771d6417014d2e45389fad1d721dfe49952ba6705f81fe466"}
Sep 06 14:14:35.840186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.839841    2576 scope.go:117] "RemoveContainer" containerID="4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"
Sep 06 14:14:35.847755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.847736    2576 scope.go:117] "RemoveContainer" containerID="4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"
Sep 06 14:14:35.848045 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:14:35.848025    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c\": container with ID starting with 4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c not found: ID does not exist" containerID="4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"
Sep 06 14:14:35.848102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.848058    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c"} err="failed to get container status \"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c\": rpc error: code = NotFound desc = could not find container \"4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c\": container with ID starting with 4aee83cd063a2f3c71e42048701d914d9a8916f19b0ff9350df7e6c2e5b8cb2c not found: ID does not exist"
Sep 06 14:14:35.855995 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.855968    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:14:35.859776 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:35.859737    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5ec585/test-model-5ec585-kserve-84c5b7cd59-mprkm"]
Sep 06 14:14:36.839325 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:36.839289    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:14:36.839538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:36.839518    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" containerID="cri-o://8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98" gracePeriod=30
Sep 06 14:14:37.178353 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178319    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"]
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178610    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98649653-bbca-4feb-8e3c-8082863ded50" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178621    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="98649653-bbca-4feb-8e3c-8082863ded50" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178628    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9774d0ca-ebea-460b-ad51-270b331abb6c" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178634    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9774d0ca-ebea-460b-ad51-270b331abb6c" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178641    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d20336-4fc9-4cf9-b901-08bd1730e327" containerName="revoke-keys"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178648    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d20336-4fc9-4cf9-b901-08bd1730e327" containerName="revoke-keys"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178662    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178667    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" containerName="main"
Sep 06 14:14:37.178723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178720    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d20336-4fc9-4cf9-b901-08bd1730e327" containerName="revoke-keys"
Sep 06 14:14:37.179049 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178728    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9774d0ca-ebea-460b-ad51-270b331abb6c" containerName="main"
Sep 06 14:14:37.179049 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178750    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="98649653-bbca-4feb-8e3c-8082863ded50" containerName="main"
Sep 06 14:14:37.179049 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.178756    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" containerName="main"
Sep 06 14:14:37.183063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.183043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:37.185057 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.185036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:14:37.185557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.185534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:14:37.187692 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.187670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"]
Sep 06 14:14:37.273663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.273629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78\" (UID: \"be8cebb7-7ac6-48a2-94dc-053843d0f513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:37.374830 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.374790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78\" (UID: \"be8cebb7-7ac6-48a2-94dc-053843d0f513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:37.375426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.375405    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78\" (UID: \"be8cebb7-7ac6-48a2-94dc-053843d0f513\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:37.494544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.494451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:37.609508 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.609476    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbfcd43f-1c39-486a-b63b-92ad6e3b3934" path="/var/lib/kubelet/pods/cbfcd43f-1c39-486a-b63b-92ad6e3b3934/volumes"
Sep 06 14:14:37.761565 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.761538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"]
Sep 06 14:14:37.763974 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:37.763947    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8cebb7_7ac6_48a2_94dc_053843d0f513.slice/crio-347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b WatchSource:0}: Error finding container 347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b: Status 404 returned error can't find the container with id 347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b
Sep 06 14:14:37.858650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:37.858608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78" event={"ID":"be8cebb7-7ac6-48a2-94dc-053843d0f513","Type":"ContainerStarted","Data":"347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b"}
Sep 06 14:14:38.197623 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:38.197585    2576 patch_prober.go:28] interesting pod/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 14:14:38.198025 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:38.197657    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 14:14:38.864337 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:38.864238    2576 generic.go:356] "Generic (PLEG): container finished" podID="be8cebb7-7ac6-48a2-94dc-053843d0f513" containerID="89578e6a1b97a0604696d94d857166a99c1604d32d12a81331694fe3389f7dd7" exitCode=0
Sep 06 14:14:38.864337 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:38.864322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78" event={"ID":"be8cebb7-7ac6-48a2-94dc-053843d0f513","Type":"ContainerDied","Data":"89578e6a1b97a0604696d94d857166a99c1604d32d12a81331694fe3389f7dd7"}
Sep 06 14:14:39.065984 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.065952    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:14:39.069221 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.069188    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.071940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.071914    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-81fae671-openshift-default-dockercfg-ttxq7\""
Sep 06 14:14:39.080701 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.080674    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:14:39.192451 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192676 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192676 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192676 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.192901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.192885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbfn\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbfn\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.294874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.295236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.295236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.294921    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.295236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.295046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.295236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.295178    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.295236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.295192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.297238 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.297215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.297461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.297442    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.302471 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.302448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbfn\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.302567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.302513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token\") pod \"e2e-mt-81fae671-openshift-default-6679d7c54d-52plk\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") " pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.380953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.380915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:39.655080 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.655052    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:14:39.657423 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:39.657390    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92191ff9_d47d_4b1a_b970_1a361a64f949.slice/crio-5657014976d8e59f6967a2a59001c9820f6c6d7714289c720354307a8be6facb WatchSource:0}: Error finding container 5657014976d8e59f6967a2a59001c9820f6c6d7714289c720354307a8be6facb: Status 404 returned error can't find the container with id 5657014976d8e59f6967a2a59001c9820f6c6d7714289c720354307a8be6facb
Sep 06 14:14:39.659746 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.659711    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:39.659820 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.659795    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:39.659867 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.659825    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:39.871200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.871166    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerStarted","Data":"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"}
Sep 06 14:14:39.871383 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.871230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerStarted","Data":"5657014976d8e59f6967a2a59001c9820f6c6d7714289c720354307a8be6facb"}
Sep 06 14:14:39.889557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.889506    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podStartSLOduration=0.88948938 podStartE2EDuration="889.48938ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:39.887592911 +0000 UTC m=+1016.913658087" watchObservedRunningTime="2026-09-06 14:14:39.88948938 +0000 UTC m=+1016.915554555"
Sep 06 14:14:39.993723 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:39.993695    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:40.101831 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.101798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca\") pod \"be8cebb7-7ac6-48a2-94dc-053843d0f513\" (UID: \"be8cebb7-7ac6-48a2-94dc-053843d0f513\") "
Sep 06 14:14:40.102169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.102143    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be8cebb7-7ac6-48a2-94dc-053843d0f513" (UID: "be8cebb7-7ac6-48a2-94dc-053843d0f513"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:14:40.203079 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.202985    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8cebb7-7ac6-48a2-94dc-053843d0f513-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:40.381254 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.381216    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:40.385977 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.385950    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:40.877219 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.877189    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"
Sep 06 14:14:40.877398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.877222    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78" event={"ID":"be8cebb7-7ac6-48a2-94dc-053843d0f513","Type":"ContainerDied","Data":"347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b"}
Sep 06 14:14:40.877398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.877273    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="347846f188b500c9b98ed2cd16e16da42cbe8ef86f81f17f91dbadde1ff03c6b"
Sep 06 14:14:40.877470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.877454    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:40.878527 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:40.878493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:42.090185 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.090158    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:14:42.223390 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223352    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223410    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpgm7\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223486    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223542    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223576    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223622    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.223978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:42.223978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223911    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:42.223978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223711    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket\") pod \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\" (UID: \"58c5e75f-1c79-469b-ab3b-b1b7e8359deb\") "
Sep 06 14:14:42.224175 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.223979    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:14:42.224175 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data" (OuterVolumeSpecName: "istio-data") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:42.224281 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224208    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:42.224571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224548    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.224653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224578    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.224653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224592    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.224653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224607    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.224653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.224622    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.225797 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.225744    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:14:42.225797 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.225779    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7" (OuterVolumeSpecName: "kube-api-access-lpgm7") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "kube-api-access-lpgm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:42.225942 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.225808    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token" (OuterVolumeSpecName: "istio-token") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:42.225942 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.225845    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58c5e75f-1c79-469b-ab3b-b1b7e8359deb" (UID: "58c5e75f-1c79-469b-ab3b-b1b7e8359deb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:42.326000 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.325962    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.326000 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.325993    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgm7\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-kube-api-access-lpgm7\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.326000 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.326006    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.326230 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.326017    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c5e75f-1c79-469b-ab3b-b1b7e8359deb-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:42.889086 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.889053    2576 generic.go:356] "Generic (PLEG): container finished" podID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerID="8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98" exitCode=0
Sep 06 14:14:42.889305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.889096    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"
Sep 06 14:14:42.889305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.889135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerDied","Data":"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"}
Sep 06 14:14:42.889305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.889186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc" event={"ID":"58c5e75f-1c79-469b-ab3b-b1b7e8359deb","Type":"ContainerDied","Data":"47cd5f13cfa69088f48931ead8c236dea0890b47336f3800d1ce10f94ccec424"}
Sep 06 14:14:42.889305 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.889208    2576 scope.go:117] "RemoveContainer" containerID="8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"
Sep 06 14:14:42.899522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.899496    2576 scope.go:117] "RemoveContainer" containerID="ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"
Sep 06 14:14:42.908446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.908427    2576 scope.go:117] "RemoveContainer" containerID="8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"
Sep 06 14:14:42.908772 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:14:42.908742    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98\": container with ID starting with 8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98 not found: ID does not exist" containerID="8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"
Sep 06 14:14:42.908848 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.908787    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98"} err="failed to get container status \"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98\": rpc error: code = NotFound desc = could not find container \"8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98\": container with ID starting with 8545a2ff248ebe99be6e8d6eba22fdda446b690e4358cdd0b823404060e97f98 not found: ID does not exist"
Sep 06 14:14:42.908848 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.908805    2576 scope.go:117] "RemoveContainer" containerID="ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"
Sep 06 14:14:42.909155 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:14:42.909119    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7\": container with ID starting with ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7 not found: ID does not exist" containerID="ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"
Sep 06 14:14:42.909276 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.909161    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7"} err="failed to get container status \"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7\": rpc error: code = NotFound desc = could not find container \"ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7\": container with ID starting with ac9db75ccc6ac506d51f2417493d886916d300465eadbe8f35f4033664c105f7 not found: ID does not exist"
Sep 06 14:14:42.910961 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.910931    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:14:42.913290 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:42.913269    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dfd6b-openshift-default-769cc4f4d-cgqmc"]
Sep 06 14:14:43.610641 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:43.610611    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" path="/var/lib/kubelet/pods/58c5e75f-1c79-469b-ab3b-b1b7e8359deb/volumes"
Sep 06 14:14:48.085675 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:48.085637    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"]
Sep 06 14:14:48.087989 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:48.087958    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ec585-qcc78"]
Sep 06 14:14:49.609476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:49.609440    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be8cebb7-7ac6-48a2-94dc-053843d0f513" path="/var/lib/kubelet/pods/be8cebb7-7ac6-48a2-94dc-053843d0f513/volumes"
Sep 06 14:14:52.781385 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.781346    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:14:52.782213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782185    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be8cebb7-7ac6-48a2-94dc-053843d0f513" containerName="revoke-keys"
Sep 06 14:14:52.782213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782211    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be8cebb7-7ac6-48a2-94dc-053843d0f513" containerName="revoke-keys"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782226    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782234    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782245    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782252    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782356    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be8cebb7-7ac6-48a2-94dc-053843d0f513" containerName="revoke-keys"
Sep 06 14:14:52.782371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782371    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.782652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.782383    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c5e75f-1c79-469b-ab3b-b1b7e8359deb" containerName="istio-proxy"
Sep 06 14:14:52.787338 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.787317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.790300 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.790193    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4e801f\"/\"openshift-service-ca.crt\""
Sep 06 14:14:52.790300 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.790206    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-4e801f\"/\"kube-root-ca.crt\""
Sep 06 14:14:52.790535 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.790518    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4e801f\"/\"noten-model-4e801f-kserve-self-signed-certs\""
Sep 06 14:14:52.790595 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.790535    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-4e801f\"/\"default-dockercfg-4q2d4\""
Sep 06 14:14:52.791560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.791538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:14:52.933846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.934040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.934040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.934040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.934040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:52.934040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:52.933992    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5w8\" (UniqueName: \"kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.034914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5w8\" (UniqueName: \"kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035539 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035605 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.035874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.035850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.037499 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.037476    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.037865 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.037847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.041643 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.041621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5w8\" (UniqueName: \"kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8\") pod \"noten-model-4e801f-kserve-8c54f956c-bxmjp\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") " pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.098309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.098265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:14:53.371341 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.371308    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:14:53.372506 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:53.372477    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd20dadec_36ee_4db4_b77b_d7defbc5a8db.slice/crio-59c10ad5ec14b863d554be399f85b67ec5fd9e7097ef97f80fbf82c2aee43ae8 WatchSource:0}: Error finding container 59c10ad5ec14b863d554be399f85b67ec5fd9e7097ef97f80fbf82c2aee43ae8: Status 404 returned error can't find the container with id 59c10ad5ec14b863d554be399f85b67ec5fd9e7097ef97f80fbf82c2aee43ae8
Sep 06 14:14:53.948053 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.948008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" event={"ID":"d20dadec-36ee-4db4-b77b-d7defbc5a8db","Type":"ContainerStarted","Data":"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"}
Sep 06 14:14:53.948481 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.948071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" event={"ID":"d20dadec-36ee-4db4-b77b-d7defbc5a8db","Type":"ContainerStarted","Data":"59c10ad5ec14b863d554be399f85b67ec5fd9e7097ef97f80fbf82c2aee43ae8"}
Sep 06 14:14:53.964004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:53.963942    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" podStartSLOduration=1.963922612 podStartE2EDuration="1.963922612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:53.963343482 +0000 UTC m=+1030.989408659" watchObservedRunningTime="2026-09-06 14:14:53.963922612 +0000 UTC m=+1030.989987788"
Sep 06 14:14:54.208508 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:54.208416    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:14:54.208743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:54.208686    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" containerID="cri-o://ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51" gracePeriod=30
Sep 06 14:14:55.037899 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.037861    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:14:55.041312 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.041284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.043413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.043384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0aaff3-openshift-default-dockercfg-nhsf8\""
Sep 06 14:14:55.053824 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.053751    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:14:55.156953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.156919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.156953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.156961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157192 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.156988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157192 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157192 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157146    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157321 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157321 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cng5\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157429 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.157429 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.157409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.258928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.258885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.258928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.258934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cng5\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.258976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.258997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259058    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.259911 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.259819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.260255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.260233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.261496 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.261474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.261833 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.261804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.266576 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.266551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.266688 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.266666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cng5\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5\") pod \"e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") " pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.340929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.340836    2576 patch_prober.go:28] interesting pod/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 14:14:55.340929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.340913    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 14:14:55.352660 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.352628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:55.634790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.634635    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:14:55.638611 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:14:55.638574    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49f66d2_13bb_4201_a8d9_d7e43a7a8543.slice/crio-cd80643896c8ef6e15718e7c3d471583e98fde2c1d0a015f88fedeaaccf74a97 WatchSource:0}: Error finding container cd80643896c8ef6e15718e7c3d471583e98fde2c1d0a015f88fedeaaccf74a97: Status 404 returned error can't find the container with id cd80643896c8ef6e15718e7c3d471583e98fde2c1d0a015f88fedeaaccf74a97
Sep 06 14:14:55.641267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.641225    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:55.641351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.641335    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:55.641392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.641375    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:55.959142 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.959111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerStarted","Data":"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"}
Sep 06 14:14:55.960090 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.959168    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerStarted","Data":"cd80643896c8ef6e15718e7c3d471583e98fde2c1d0a015f88fedeaaccf74a97"}
Sep 06 14:14:55.982651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:55.982589    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podStartSLOduration=0.982568233 podStartE2EDuration="982.568233ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:14:55.979693557 +0000 UTC m=+1033.005758742" watchObservedRunningTime="2026-09-06 14:14:55.982568233 +0000 UTC m=+1033.008633409"
Sep 06 14:14:56.028955 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.028917    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:40956->10.133.0.43:15021: read: connection reset by peer" start-of-body=
Sep 06 14:14:56.029094 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.028989    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:40956->10.133.0.43:15021: read: connection reset by peer"
Sep 06 14:14:56.353550 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.353508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:56.358674 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.358644    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:56.965052 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.965017    2576 generic.go:356] "Generic (PLEG): container finished" podID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerID="62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198" exitCode=0
Sep 06 14:14:56.965231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.965089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerDied","Data":"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"}
Sep 06 14:14:56.965444 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.965422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:56.965640 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.965622    2576 scope.go:117] "RemoveContainer" containerID="62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"
Sep 06 14:14:56.966518 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.966474    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:14:56.966739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.966708    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:56.966833 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.966790    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:56.966833 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:56.966821    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:14:57.971370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:57.971334    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerStarted","Data":"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"}
Sep 06 14:14:58.381489 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:58.381377    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:14:58.381656 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:58.381607    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 06 14:14:58.381707 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:58.381668    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 06 14:14:59.381615 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.381557    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 06 14:14:59.382096 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.381622    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 06 14:14:59.489007 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.488947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:14:59.603975 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.603939    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604164 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604001    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkfqv\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604164 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604035    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604164 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604092    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604164 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert\") pod \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\" (UID: \"a09a9b77-a411-4f5d-8915-03a7be0f3adc\") "
Sep 06 14:14:59.604391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604372    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data" (OuterVolumeSpecName: "istio-data") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:59.604790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604629    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.604790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604653    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.604790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604721    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:59.604790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.604739    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:59.605161 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.605110    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:14:59.606744 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.606706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv" (OuterVolumeSpecName: "kube-api-access-dkfqv") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "kube-api-access-dkfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:59.607047 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.607023    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token" (OuterVolumeSpecName: "istio-token") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:14:59.607189 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.607023    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:14:59.607284 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.607033    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a09a9b77-a411-4f5d-8915-03a7be0f3adc" (UID: "a09a9b77-a411-4f5d-8915-03a7be0f3adc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:14:59.705560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705525    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705556    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705565    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705840 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705577    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705840 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705591    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfqv\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-kube-api-access-dkfqv\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705840 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705601    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.705840 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.705610    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a09a9b77-a411-4f5d-8915-03a7be0f3adc-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:14:59.982573 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.982539    2576 generic.go:356] "Generic (PLEG): container finished" podID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerID="ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51" exitCode=0
Sep 06 14:14:59.982747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.982592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"
Sep 06 14:14:59.982747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.982607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerDied","Data":"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"}
Sep 06 14:14:59.982747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.982649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd" event={"ID":"a09a9b77-a411-4f5d-8915-03a7be0f3adc","Type":"ContainerDied","Data":"b7cb2ce91859db0eacc4f2fef252680630dd5367ecfe6d9a395dc017d6f6707b"}
Sep 06 14:14:59.982747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.982670    2576 scope.go:117] "RemoveContainer" containerID="ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"
Sep 06 14:14:59.991596 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.991574    2576 scope.go:117] "RemoveContainer" containerID="62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"
Sep 06 14:14:59.997200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:14:59.997175    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:15:00.000619 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.000594    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ec585-openshift-default-5664665974-7dlsd"]
Sep 06 14:15:00.000836 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.000821    2576 scope.go:117] "RemoveContainer" containerID="ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"
Sep 06 14:15:00.001164 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:00.001138    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51\": container with ID starting with ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51 not found: ID does not exist" containerID="ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"
Sep 06 14:15:00.001259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.001170    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51"} err="failed to get container status \"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51\": rpc error: code = NotFound desc = could not find container \"ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51\": container with ID starting with ac3a25c410464cccd25a95649d7bd401bfa116cd536676e653f81471bf9e5e51 not found: ID does not exist"
Sep 06 14:15:00.001259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.001189    2576 scope.go:117] "RemoveContainer" containerID="62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"
Sep 06 14:15:00.001514 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:00.001490    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847\": container with ID starting with 62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847 not found: ID does not exist" containerID="62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"
Sep 06 14:15:00.001556 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.001521    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847"} err="failed to get container status \"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847\": rpc error: code = NotFound desc = could not find container \"62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847\": container with ID starting with 62dba8370374eed60e91d0fe0af17303cbe8993c3484766d26d97d4652551847 not found: ID does not exist"
Sep 06 14:15:00.133986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.133942    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:00.134483 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.134456    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.134483 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.134482    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.134678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.134590    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.134678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.134605    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.137469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.137426    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"]
Sep 06 14:15:00.137730 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.137711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"
Sep 06 14:15:00.137905 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.137883    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.137905 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.137901    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" containerName="istio-proxy"
Sep 06 14:15:00.139572 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.139553    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"
Sep 06 14:15:00.139978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.139957    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:15:00.146727 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.146067    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:00.148311 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.148281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"]
Sep 06 14:15:00.151888 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.151866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"
Sep 06 14:15:00.159512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.159490    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"
Sep 06 14:15:00.382561 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.382518    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 06 14:15:00.382988 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.382605    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 06 14:15:00.444963 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.442850    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:00.450446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.447733    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"]
Sep 06 14:15:00.989630 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.989594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt" event={"ID":"3b3869b8-f100-4573-ae50-8028daa257a7","Type":"ContainerStarted","Data":"28dd2a43cd2c6f957b42da70df151f80f0e59d530cea30614c033fbee20aa76e"}
Sep 06 14:15:00.989832 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.989649    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt" event={"ID":"3b3869b8-f100-4573-ae50-8028daa257a7","Type":"ContainerStarted","Data":"22d57ff23d26648d7da1f16a027b546ae719271840ee0284088346cb5eb32a97"}
Sep 06 14:15:00.991218 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.991194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerStarted","Data":"d199131ef0f9fd7705c5b44f640451571a27c49e2f81cae7ef77cf4f6e253d8c"}
Sep 06 14:15:00.991303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:00.991235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerStarted","Data":"0471e5d0d6b1172b72c173167eec690d9e807e6475fce820291a897cc6826df2"}
Sep 06 14:15:01.002344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.002298    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt" podStartSLOduration=1.002282542 podStartE2EDuration="1.002282542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:01.00154447 +0000 UTC m=+1038.027609648" watchObservedRunningTime="2026-09-06 14:15:01.002282542 +0000 UTC m=+1038.028347716"
Sep 06 14:15:01.013843 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.013726    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" podStartSLOduration=1.013706284 podStartE2EDuration="1.013706284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:01.013199105 +0000 UTC m=+1038.039264273" watchObservedRunningTime="2026-09-06 14:15:01.013706284 +0000 UTC m=+1038.039771460"
Sep 06 14:15:01.382103 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.382062    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 06 14:15:01.382293 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.382121    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 06 14:15:01.610176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.610141    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a09a9b77-a411-4f5d-8915-03a7be0f3adc" path="/var/lib/kubelet/pods/a09a9b77-a411-4f5d-8915-03a7be0f3adc/volumes"
Sep 06 14:15:01.997165 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.997131    2576 generic.go:356] "Generic (PLEG): container finished" podID="3b3869b8-f100-4573-ae50-8028daa257a7" containerID="28dd2a43cd2c6f957b42da70df151f80f0e59d530cea30614c033fbee20aa76e" exitCode=0
Sep 06 14:15:01.997348 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.997221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt" event={"ID":"3b3869b8-f100-4573-ae50-8028daa257a7","Type":"ContainerDied","Data":"28dd2a43cd2c6f957b42da70df151f80f0e59d530cea30614c033fbee20aa76e"}
Sep 06 14:15:01.998675 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.998653    2576 generic.go:356] "Generic (PLEG): container finished" podID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerID="d199131ef0f9fd7705c5b44f640451571a27c49e2f81cae7ef77cf4f6e253d8c" exitCode=7
Sep 06 14:15:01.998797 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.998691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerDied","Data":"d199131ef0f9fd7705c5b44f640451571a27c49e2f81cae7ef77cf4f6e253d8c"}
Sep 06 14:15:01.998932 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:01.998916    2576 scope.go:117] "RemoveContainer" containerID="d199131ef0f9fd7705c5b44f640451571a27c49e2f81cae7ef77cf4f6e253d8c"
Sep 06 14:15:02.382136 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:02.382094    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 06 14:15:02.382291 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:02.382160    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 06 14:15:03.005355 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.005322    2576 generic.go:356] "Generic (PLEG): container finished" podID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerID="9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f" exitCode=7
Sep 06 14:15:03.005791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.005410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerDied","Data":"9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f"}
Sep 06 14:15:03.005791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.005478    2576 scope.go:117] "RemoveContainer" containerID="d199131ef0f9fd7705c5b44f640451571a27c49e2f81cae7ef77cf4f6e253d8c"
Sep 06 14:15:03.005791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.005565    2576 scope.go:117] "RemoveContainer" containerID="9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f"
Sep 06 14:15:03.005973 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:03.005837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk_odh-ai-gateway-infra(a22d72ec-ef41-425d-abdd-4aa31971f57c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c"
Sep 06 14:15:03.098725 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.098690    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:03.098939 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.098738    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:03.127462 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.127171    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:03.178246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.178221    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"
Sep 06 14:15:03.466977 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.466925    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:15:03.467420 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.467397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:15:03.468614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:03.468588    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:15:04.012342 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:04.012310    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt"
Sep 06 14:15:04.012835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:04.012337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811735-6ncxt" event={"ID":"3b3869b8-f100-4573-ae50-8028daa257a7","Type":"ContainerDied","Data":"22d57ff23d26648d7da1f16a027b546ae719271840ee0284088346cb5eb32a97"}
Sep 06 14:15:04.012835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:04.012421    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22d57ff23d26648d7da1f16a027b546ae719271840ee0284088346cb5eb32a97"
Sep 06 14:15:04.014160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:04.014133    2576 scope.go:117] "RemoveContainer" containerID="9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f"
Sep 06 14:15:04.014436 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:04.014407    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk_odh-ai-gateway-infra(a22d72ec-ef41-425d-abdd-4aa31971f57c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c"
Sep 06 14:15:04.025883 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:04.025851    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:13.006962 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:13.006908    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:15:13.007378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:13.007004    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:15:14.074044 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.074011    2576 generic.go:356] "Generic (PLEG): container finished" podID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerID="71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda" exitCode=0
Sep 06 14:15:14.074480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.074084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerDied","Data":"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"}
Sep 06 14:15:14.074537 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.074489    2576 scope.go:117] "RemoveContainer" containerID="71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"
Sep 06 14:15:14.075498 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.075467    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:14.075556 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.075534    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:14.075597 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:14.075570    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:15.080561 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:15.080521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerStarted","Data":"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"}
Sep 06 14:15:15.353416 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:15.353319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:15:15.353635 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:15.353610    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 06 14:15:15.353686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:15.353662    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 06 14:15:16.353888 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:16.353841    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 06 14:15:16.354342 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:16.353900    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 06 14:15:17.354086 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:17.354047    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 06 14:15:17.354482 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:17.354108    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 06 14:15:17.604652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:17.604569    2576 scope.go:117] "RemoveContainer" containerID="9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f"
Sep 06 14:15:18.097237 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:18.097200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerStarted","Data":"96eda92c8a933fab02cb3be0bb6025fb54f2248bb93108d3805841c8b8fc3830"}
Sep 06 14:15:18.353754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:18.353648    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 06 14:15:18.353754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:18.353716    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 06 14:15:18.804860 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:18.804827    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:15:19.103827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.103716    2576 generic.go:356] "Generic (PLEG): container finished" podID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerID="96eda92c8a933fab02cb3be0bb6025fb54f2248bb93108d3805841c8b8fc3830" exitCode=0
Sep 06 14:15:19.103827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.103794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerDied","Data":"96eda92c8a933fab02cb3be0bb6025fb54f2248bb93108d3805841c8b8fc3830"}
Sep 06 14:15:19.104110 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.103861    2576 scope.go:117] "RemoveContainer" containerID="9192822ebfcc6f4c71b6df3adc60f6849af36a8caaa5fcdf2210c140ea89905f"
Sep 06 14:15:19.126918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.126881    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:19.354019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.353927    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 06 14:15:19.354019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:19.354002    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 06 14:15:20.279279 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:20.279243    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"
Sep 06 14:15:20.564110 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:20.564076    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:15:20.564586 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:20.564557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:15:20.567048 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:20.567028    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:15:21.115262 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.115227    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"
Sep 06 14:15:21.115547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.115263    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk" event={"ID":"a22d72ec-ef41-425d-abdd-4aa31971f57c","Type":"ContainerDied","Data":"0471e5d0d6b1172b72c173167eec690d9e807e6475fce820291a897cc6826df2"}
Sep 06 14:15:21.115547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.115313    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0471e5d0d6b1172b72c173167eec690d9e807e6475fce820291a897cc6826df2"
Sep 06 14:15:21.144482 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.144453    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:21.150005 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.149977    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0aaff3-29811735-85gzk"]
Sep 06 14:15:21.609624 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:21.609593    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" path="/var/lib/kubelet/pods/a22d72ec-ef41-425d-abdd-4aa31971f57c/volumes"
Sep 06 14:15:23.313592 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.313554    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:15:23.314037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314021    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.314082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314040    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.314082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314061    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3869b8-f100-4573-ae50-8028daa257a7" containerName="cleanup"
Sep 06 14:15:23.314082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314070    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3869b8-f100-4573-ae50-8028daa257a7" containerName="cleanup"
Sep 06 14:15:23.314178 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314081    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.314178 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314091    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.314178 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314170    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b3869b8-f100-4573-ae50-8028daa257a7" containerName="cleanup"
Sep 06 14:15:23.314271 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314180    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.314271 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.314190    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:23.319911 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.318289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.322508 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.322449    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c443d8-openshift-default-dockercfg-qddqp\""
Sep 06 14:15:23.326489 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.326462    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:15:23.421167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.421435 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.421430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rv6\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.522875 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.522875 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.522979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523084    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rv6\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523290    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.523886 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.523865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.525332 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.525304    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.525722 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.525705    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.530684 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.530656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.530882 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.530865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rv6\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6\") pod \"e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") " pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.634030 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.633948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:23.916337 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.916313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:15:23.918418 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:15:23.918386    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28489758_86e0_493b_bb12_8e0ee99366da.slice/crio-8b6907772c377d13e97b8d3d07ccf103090149be2f08ee381bfb792ec162b4af WatchSource:0}: Error finding container 8b6907772c377d13e97b8d3d07ccf103090149be2f08ee381bfb792ec162b4af: Status 404 returned error can't find the container with id 8b6907772c377d13e97b8d3d07ccf103090149be2f08ee381bfb792ec162b4af
Sep 06 14:15:23.920998 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.920953    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:23.921117 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.921049    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:23.921117 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:23.921095    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:24.130730 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:24.130691    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerStarted","Data":"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"}
Sep 06 14:15:24.130912 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:24.130756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerStarted","Data":"8b6907772c377d13e97b8d3d07ccf103090149be2f08ee381bfb792ec162b4af"}
Sep 06 14:15:24.149839 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:24.149787    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podStartSLOduration=1.1497713649999999 podStartE2EDuration="1.149771365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:24.147485648 +0000 UTC m=+1061.173550822" watchObservedRunningTime="2026-09-06 14:15:24.149771365 +0000 UTC m=+1061.175836530"
Sep 06 14:15:24.634565 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:24.634522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:24.639604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:24.639579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:25.136140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:25.136115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:25.137186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:25.137166    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:30.780540 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:30.780490    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:15:30.781038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:30.780721    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" containerID="cri-o://74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9" gracePeriod=30
Sep 06 14:15:33.470357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:33.470320    2576 patch_prober.go:28] interesting pod/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 14:15:33.470718 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:33.470380    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 14:15:33.850246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:33.850131    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerName="main" containerID="cri-o://97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0" gracePeriod=45
Sep 06 14:15:34.014618 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.014572    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused"
Sep 06 14:15:34.102656 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.102591    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:34.178666 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.178636    2576 generic.go:356] "Generic (PLEG): container finished" podID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerID="97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0" exitCode=0
Sep 06 14:15:34.178853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.178680    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"
Sep 06 14:15:34.178853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.178713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" event={"ID":"d20dadec-36ee-4db4-b77b-d7defbc5a8db","Type":"ContainerDied","Data":"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"}
Sep 06 14:15:34.178853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.178748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp" event={"ID":"d20dadec-36ee-4db4-b77b-d7defbc5a8db","Type":"ContainerDied","Data":"59c10ad5ec14b863d554be399f85b67ec5fd9e7097ef97f80fbf82c2aee43ae8"}
Sep 06 14:15:34.178853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.178788    2576 scope.go:117] "RemoveContainer" containerID="97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"
Sep 06 14:15:34.186806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.186780    2576 scope.go:117] "RemoveContainer" containerID="97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"
Sep 06 14:15:34.187081 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:34.187063    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0\": container with ID starting with 97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0 not found: ID does not exist" containerID="97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"
Sep 06 14:15:34.187139 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.187091    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0"} err="failed to get container status \"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0\": rpc error: code = NotFound desc = could not find container \"97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0\": container with ID starting with 97c736299872aa3ddb8bcd9c314a0feae576291f5fa9da7318dac03e41b97ad0 not found: ID does not exist"
Sep 06 14:15:34.226718 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226670    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.226718 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.226920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.226920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226817    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.226920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v5w8\" (UniqueName: \"kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.226920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.226913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs\") pod \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\" (UID: \"d20dadec-36ee-4db4-b77b-d7defbc5a8db\") "
Sep 06 14:15:34.227099 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:34.227160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227095    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache" (OuterVolumeSpecName: "model-cache") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:34.227160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227109    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home" (OuterVolumeSpecName: "home") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:34.227299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227270    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.227299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227288    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.227299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.227297    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.229153 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.229124    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm" (OuterVolumeSpecName: "dshm") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:34.229240 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.229174    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8" (OuterVolumeSpecName: "kube-api-access-2v5w8") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "kube-api-access-2v5w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:15:34.229240 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.229176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d20dadec-36ee-4db4-b77b-d7defbc5a8db" (UID: "d20dadec-36ee-4db4-b77b-d7defbc5a8db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:15:34.328652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.328612    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20dadec-36ee-4db4-b77b-d7defbc5a8db-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.328652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.328646    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20dadec-36ee-4db4-b77b-d7defbc5a8db-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.328652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.328655    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5w8\" (UniqueName: \"kubernetes.io/projected/d20dadec-36ee-4db4-b77b-d7defbc5a8db-kube-api-access-2v5w8\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:34.499074 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.499040    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:15:34.502364 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:34.502338    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-4e801f/noten-model-4e801f-kserve-8c54f956c-bxmjp"]
Sep 06 14:15:35.610401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:35.610364    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" path="/var/lib/kubelet/pods/d20dadec-36ee-4db4-b77b-d7defbc5a8db/volumes"
Sep 06 14:15:36.060733 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.060710    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:15:36.146541 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146584    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146609    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146640    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146667    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146702    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.146749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.147143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146784    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zbfn\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn\") pod \"92191ff9-d47d-4b1a-b970-1a361a64f949\" (UID: \"92191ff9-d47d-4b1a-b970-1a361a64f949\") "
Sep 06 14:15:36.147143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data" (OuterVolumeSpecName: "istio-data") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:36.147143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146934    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:36.147143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.146957    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:36.147143 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:36.147386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:15:36.147386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147217    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.147386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147237    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.147386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147251    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.147386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.147264    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.148979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.148920    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token" (OuterVolumeSpecName: "istio-token") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:15:36.149083 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.148996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:15:36.149083 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.148993    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn" (OuterVolumeSpecName: "kube-api-access-6zbfn") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "kube-api-access-6zbfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:15:36.149083 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.149073    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92191ff9-d47d-4b1a-b970-1a361a64f949" (UID: "92191ff9-d47d-4b1a-b970-1a361a64f949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:15:36.191772 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.191726    2576 generic.go:356] "Generic (PLEG): container finished" podID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerID="74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9" exitCode=0
Sep 06 14:15:36.191923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.191786    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"
Sep 06 14:15:36.191923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.191797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerDied","Data":"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"}
Sep 06 14:15:36.191923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.191840    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk" event={"ID":"92191ff9-d47d-4b1a-b970-1a361a64f949","Type":"ContainerDied","Data":"5657014976d8e59f6967a2a59001c9820f6c6d7714289c720354307a8be6facb"}
Sep 06 14:15:36.191923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.191862    2576 scope.go:117] "RemoveContainer" containerID="74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"
Sep 06 14:15:36.201313 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.201289    2576 scope.go:117] "RemoveContainer" containerID="62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"
Sep 06 14:15:36.209030 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.209013    2576 scope.go:117] "RemoveContainer" containerID="74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"
Sep 06 14:15:36.209330 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:36.209306    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9\": container with ID starting with 74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9 not found: ID does not exist" containerID="74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"
Sep 06 14:15:36.209399 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.209338    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9"} err="failed to get container status \"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9\": rpc error: code = NotFound desc = could not find container \"74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9\": container with ID starting with 74b7d97c607bde576eb53d74200d6fee2ceaf6df335511ff6a2a8631afda14e9 not found: ID does not exist"
Sep 06 14:15:36.209399 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.209356    2576 scope.go:117] "RemoveContainer" containerID="62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"
Sep 06 14:15:36.209591 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:15:36.209573    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198\": container with ID starting with 62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198 not found: ID does not exist" containerID="62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"
Sep 06 14:15:36.209632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.209596    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198"} err="failed to get container status \"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198\": rpc error: code = NotFound desc = could not find container \"62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198\": container with ID starting with 62dddfd4a38049f5874c12832e62e378612e807eda8bf94d82d3ed86a7ae2198 not found: ID does not exist"
Sep 06 14:15:36.213541 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.213490    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:15:36.232005 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.231928    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-81fae671-openshift-default-6679d7c54d-52plk"]
Sep 06 14:15:36.248392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.248356    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zbfn\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-kube-api-access-6zbfn\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.248392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.248387    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.248392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.248400    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.248633 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.248409    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92191ff9-d47d-4b1a-b970-1a361a64f949-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:36.248633 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:36.248417    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92191ff9-d47d-4b1a-b970-1a361a64f949-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:15:37.609731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:37.609697    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" path="/var/lib/kubelet/pods/92191ff9-d47d-4b1a-b970-1a361a64f949/volumes"
Sep 06 14:15:44.056584 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056495    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056928    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056943    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056970    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056978    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.056999    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerName="main"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057007    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerName="main"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057016    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057024    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057086    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22d72ec-ef41-425d-abdd-4aa31971f57c" containerName="cleanup"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057096    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057106    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20dadec-36ee-4db4-b77b-d7defbc5a8db" containerName="main"
Sep 06 14:15:44.057120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.057115    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="92191ff9-d47d-4b1a-b970-1a361a64f949" containerName="istio-proxy"
Sep 06 14:15:44.060303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.060286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.063024 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.063002    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6328f9-openshift-default-dockercfg-l55pw\""
Sep 06 14:15:44.071689 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.071635    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:15:44.219407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtvm\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219544    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219612 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.219869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.219728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.234394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.234368    2576 generic.go:356] "Generic (PLEG): container finished" podID="28489758-86e0-493b-bb12-8e0ee99366da" containerID="16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a" exitCode=0
Sep 06 14:15:44.234551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.234446    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerDied","Data":"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"}
Sep 06 14:15:44.234885 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.234870    2576 scope.go:117] "RemoveContainer" containerID="16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"
Sep 06 14:15:44.235995 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.235968    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:44.236081 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.236038    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:44.236081 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.236072    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:44.320638 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.320796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.320796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtvm\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.320928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.320928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.320964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321038 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321203 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321421 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321421 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321380    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321574 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321634    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.321867 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.321813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.323666 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.323599    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.324100 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.324079    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.338455 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.338425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.338707 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.338683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtvm\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm\") pod \"e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") " pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.372306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.372270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:44.653521 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.653215    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:15:44.656258 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:15:44.656225    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a86078c_c84e_4ddc_8eba_3b03dd640531.slice/crio-1622f23e3c741f88939359e8420e96ca301eb8615b934f8065bfff2228a52318 WatchSource:0}: Error finding container 1622f23e3c741f88939359e8420e96ca301eb8615b934f8065bfff2228a52318: Status 404 returned error can't find the container with id 1622f23e3c741f88939359e8420e96ca301eb8615b934f8065bfff2228a52318
Sep 06 14:15:44.658902 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.658870    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:44.658993 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.658955    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:44.659064 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:44.659000    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:15:45.242093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.242048    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerStarted","Data":"7b3fdeb937ba916bbc1b7880501b93c8747574a138212e3f85ff84fa5284f226"}
Sep 06 14:15:45.242564 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.242114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerStarted","Data":"1622f23e3c741f88939359e8420e96ca301eb8615b934f8065bfff2228a52318"}
Sep 06 14:15:45.244027 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.244004    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerStarted","Data":"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"}
Sep 06 14:15:45.260115 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.260064    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podStartSLOduration=1.260046826 podStartE2EDuration="1.260046826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:45.258561 +0000 UTC m=+1082.284626176" watchObservedRunningTime="2026-09-06 14:15:45.260046826 +0000 UTC m=+1082.286112000"
Sep 06 14:15:45.373473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.373435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:45.379126 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.379096    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:45.635234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.635125    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:45.635685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.635638    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 06 14:15:45.635835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:45.635722    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 06 14:15:46.248470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:46.248427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:46.249560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:46.249542    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:15:46.635072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:46.634965    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 06 14:15:46.635363 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:46.635331    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 06 14:15:47.634673 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:47.634616    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 06 14:15:47.635219 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:47.634693    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 06 14:15:48.635910 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:48.635868    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 06 14:15:48.636856 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:48.636447    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 06 14:15:49.634813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:49.634748    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 06 14:15:49.635043 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:49.634840    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 06 14:15:50.831747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:50.831710    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:50.832220 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:50.832039    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:50.833858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:50.833830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:15:53.591346 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.591313    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:15:53.595032 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.595014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.598121 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.598070    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"openshift-service-ca.crt\""
Sep 06 14:15:53.598121 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.598096    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"default-dockercfg-lnc9k\""
Sep 06 14:15:53.598121 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.598115    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"kube-root-ca.crt\""
Sep 06 14:15:53.598341 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.598118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"auth-test-model-0aaff3-kserve-self-signed-certs\""
Sep 06 14:15:53.603918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.603665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.603918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.603796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.603918 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.603871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8zj\" (UniqueName: \"kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.604156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.603961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.604156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.604000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.604156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.604130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.614626 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.614599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:15:53.705895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.705840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.705895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.705895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.705921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8zj\" (UniqueName: \"kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.705967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.705982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.706026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706415 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.706303    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706415 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.706392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.706605 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.706585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.708237 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.708219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.708483 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.708465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.712755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.712732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8zj\" (UniqueName: \"kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj\") pod \"auth-test-model-0aaff3-kserve-6fbf954d95-9gv24\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:53.906060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:53.905962    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:15:54.176119 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:54.176091    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:15:54.178406 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:15:54.178370    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e3d12a3_4756_4a15_8478_3ea08e7a45a7.slice/crio-5e6777605cba4b026799c4ce2b9b1c1b393960ebc2d5c920624a5dbfa98b37c6 WatchSource:0}: Error finding container 5e6777605cba4b026799c4ce2b9b1c1b393960ebc2d5c920624a5dbfa98b37c6: Status 404 returned error can't find the container with id 5e6777605cba4b026799c4ce2b9b1c1b393960ebc2d5c920624a5dbfa98b37c6
Sep 06 14:15:54.285828 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:54.285792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" event={"ID":"4e3d12a3-4756-4a15-8478-3ea08e7a45a7","Type":"ContainerStarted","Data":"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"}
Sep 06 14:15:54.285989 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:54.285837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" event={"ID":"4e3d12a3-4756-4a15-8478-3ea08e7a45a7","Type":"ContainerStarted","Data":"5e6777605cba4b026799c4ce2b9b1c1b393960ebc2d5c920624a5dbfa98b37c6"}
Sep 06 14:15:54.304256 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:15:54.303669    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" podStartSLOduration=1.30364733 podStartE2EDuration="1.30364733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:15:54.30171108 +0000 UTC m=+1091.327776262" watchObservedRunningTime="2026-09-06 14:15:54.30364733 +0000 UTC m=+1091.329712507"
Sep 06 14:16:02.288973 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:02.288920    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:16:02.289356 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:02.288994    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:16:03.906235 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:03.906147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:16:03.906235 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:03.906198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:16:03.918232 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:03.918202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:16:04.342442 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.342408    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerID="7b3fdeb937ba916bbc1b7880501b93c8747574a138212e3f85ff84fa5284f226" exitCode=0
Sep 06 14:16:04.342645 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.342496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerDied","Data":"7b3fdeb937ba916bbc1b7880501b93c8747574a138212e3f85ff84fa5284f226"}
Sep 06 14:16:04.343054 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.343029    2576 scope.go:117] "RemoveContainer" containerID="7b3fdeb937ba916bbc1b7880501b93c8747574a138212e3f85ff84fa5284f226"
Sep 06 14:16:04.344276 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.344248    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:04.344341 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.344321    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:04.344378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.344353    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:04.354707 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:04.354683    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:16:05.349520 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:05.349482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerStarted","Data":"1ffadb8ebfd5c5ad8717f80645894e76738a91469ec56a7d962b5c559700bc19"}
Sep 06 14:16:05.372441 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:05.372404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:16:05.372657 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:05.372640    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:05.372701 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:05.372679    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:06.372904 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:06.372870    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:06.373299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:06.372923    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:07.372992 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:07.372958    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:07.373383 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:07.373011    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:08.373062 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:08.373022    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:08.373539 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:08.373084    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:09.372909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:09.372874    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:09.373094 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:09.372940    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:10.372770 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:10.372716    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 06 14:16:10.372968 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:10.372806    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 06 14:16:11.377192 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:11.377157    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:16:11.378173 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:11.377942    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:16:11.380803 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:11.380781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:16:12.547477 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.547441    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:16:12.550977 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.550955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.553344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.553319    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a388e1-openshift-default-dockercfg-hgblc\""
Sep 06 14:16:12.561050 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.561024    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:16:12.694102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694053    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694112    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694481 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppldm\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694481 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694481 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694598 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.694638 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.694622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppldm\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796662 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796719 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796719 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.796868 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.796750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.797631 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.797404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.804650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.799164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.804650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.802923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.806407 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.805527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.807454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.807417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppldm\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm\") pod \"e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") " pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:12.862021 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:12.861984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:13.160315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.160285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:16:13.161991 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:16:13.161963    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d66be43_19c7_4a6b_be6a_26870e65cc43.slice/crio-62396bbb5dbe46c50e123ec835023e6b4958d80de2e481a98d761ba3dafadbed WatchSource:0}: Error finding container 62396bbb5dbe46c50e123ec835023e6b4958d80de2e481a98d761ba3dafadbed: Status 404 returned error can't find the container with id 62396bbb5dbe46c50e123ec835023e6b4958d80de2e481a98d761ba3dafadbed
Sep 06 14:16:13.164384 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.164340    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:13.164492 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.164427    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:13.164492 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.164470    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:13.395176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.395130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerStarted","Data":"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"}
Sep 06 14:16:13.395370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.395200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerStarted","Data":"62396bbb5dbe46c50e123ec835023e6b4958d80de2e481a98d761ba3dafadbed"}
Sep 06 14:16:13.413715 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.413671    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podStartSLOduration=1.4136535860000001 podStartE2EDuration="1.413653586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:13.412207128 +0000 UTC m=+1110.438272327" watchObservedRunningTime="2026-09-06 14:16:13.413653586 +0000 UTC m=+1110.439718760"
Sep 06 14:16:13.862736 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:13.862624    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:14.867236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:14.867208    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:15.404905 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:15.404871    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:15.405888 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:15.405869    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:28.791958 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.791917    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:16:28.796186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.796163    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.798781 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.798744    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"kube-root-ca.crt\""
Sep 06 14:16:28.800273 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.800066    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"openshift-service-ca.crt\""
Sep 06 14:16:28.800273 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.800093    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"default-dockercfg-55pl5\""
Sep 06 14:16:28.800449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.800401    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"auth-test-model-c443d8-kserve-self-signed-certs\""
Sep 06 14:16:28.802599 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.802574    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:16:28.953059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.953059 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.953280 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.953280 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953190    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkrg\" (UniqueName: \"kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.953280 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:28.953378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:28.953310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054631 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054631 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkrg\" (UniqueName: \"kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054631 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054830 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054830 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.054996 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.054975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.055149 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.055125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.056637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.056618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.056945 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.056927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.061979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.061956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkrg\" (UniqueName: \"kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg\") pod \"auth-test-model-c443d8-kserve-6b8d98675f-6htt4\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.107254 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.106971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:29.402134 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.402106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:16:29.404066 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:16:29.404037    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod390b4051_fda5_44f9_ab68_6df9f6ae4621.slice/crio-d0b86b58b75ca41c4ee47b21631d723f0d9104188bbe74b22e2793aa16c10fed WatchSource:0}: Error finding container d0b86b58b75ca41c4ee47b21631d723f0d9104188bbe74b22e2793aa16c10fed: Status 404 returned error can't find the container with id d0b86b58b75ca41c4ee47b21631d723f0d9104188bbe74b22e2793aa16c10fed
Sep 06 14:16:29.479633 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:29.479591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" event={"ID":"390b4051-fda5-44f9-ab68-6df9f6ae4621","Type":"ContainerStarted","Data":"d0b86b58b75ca41c4ee47b21631d723f0d9104188bbe74b22e2793aa16c10fed"}
Sep 06 14:16:30.486328 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:30.486291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" event={"ID":"390b4051-fda5-44f9-ab68-6df9f6ae4621","Type":"ContainerStarted","Data":"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"}
Sep 06 14:16:30.502226 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:30.502168    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" podStartSLOduration=2.502151144 podStartE2EDuration="2.502151144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:16:30.500472415 +0000 UTC m=+1127.526537591" watchObservedRunningTime="2026-09-06 14:16:30.502151144 +0000 UTC m=+1127.528216315"
Sep 06 14:16:33.503674 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.503637    2576 generic.go:356] "Generic (PLEG): container finished" podID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerID="cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070" exitCode=0
Sep 06 14:16:33.504112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.503714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerDied","Data":"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"}
Sep 06 14:16:33.504184 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.504169    2576 scope.go:117] "RemoveContainer" containerID="cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"
Sep 06 14:16:33.505274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.505242    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:33.505367 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.505311    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:33.505367 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:33.505343    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:16:34.509798 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:34.509742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerStarted","Data":"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"}
Sep 06 14:16:34.862269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:34.862174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:34.862430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:34.862407    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:34.862479 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:34.862460    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:35.863437 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:35.863400    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:35.863896 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:35.863490    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:36.863137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:36.863095    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:36.863327 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:36.863167    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:37.862615 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:37.862576    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:37.863048 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:37.862654    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:38.862730 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:38.862693    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:38.863145 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:38.862783    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:39.108108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.108063    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:39.108317 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.108146    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:39.144147 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.144060    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:39.555779 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.555740    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:16:39.862504 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.862405    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 06 14:16:39.862504 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:39.862467    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 06 14:16:40.823978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.823943    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:40.827008 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.826990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:40.829472 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.829435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\""
Sep 06 14:16:40.831117 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.831087    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:40.866899 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.866866    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:40.867202 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.867183    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:40.867878 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.867857    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:16:40.874072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.874052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wm62\" (UniqueName: \"kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62\") pod \"test-curl-7542-140355587229136\" (UID: \"1289ffd6-90c3-4a0d-8c91-3d996d37e55e\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:40.974617 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.974572    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wm62\" (UniqueName: \"kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62\") pod \"test-curl-7542-140355587229136\" (UID: \"1289ffd6-90c3-4a0d-8c91-3d996d37e55e\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:40.982460 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:40.982438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wm62\" (UniqueName: \"kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62\") pod \"test-curl-7542-140355587229136\" (UID: \"1289ffd6-90c3-4a0d-8c91-3d996d37e55e\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:41.137331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:41.137235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:41.407827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:41.407731    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:41.411295 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:16:41.411266    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1289ffd6_90c3_4a0d_8c91_3d996d37e55e.slice/crio-df3a0a1af8fdb01ea70a8207ca4cdf985ec5abc115ff5d8d36e4c9d939df3e04 WatchSource:0}: Error finding container df3a0a1af8fdb01ea70a8207ca4cdf985ec5abc115ff5d8d36e4c9d939df3e04: Status 404 returned error can't find the container with id df3a0a1af8fdb01ea70a8207ca4cdf985ec5abc115ff5d8d36e4c9d939df3e04
Sep 06 14:16:41.412654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:41.412633    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 14:16:41.555264 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:41.555223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229136" event={"ID":"1289ffd6-90c3-4a0d-8c91-3d996d37e55e","Type":"ContainerStarted","Data":"df3a0a1af8fdb01ea70a8207ca4cdf985ec5abc115ff5d8d36e4c9d939df3e04"}
Sep 06 14:16:42.561403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:42.561364    2576 generic.go:356] "Generic (PLEG): container finished" podID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" containerID="324bc66a4c9dc4823e4d52c89aa05711bca072dab59f3e707e96592f6cd0cefb" exitCode=0
Sep 06 14:16:42.561822 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:42.561452    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229136" event={"ID":"1289ffd6-90c3-4a0d-8c91-3d996d37e55e","Type":"ContainerDied","Data":"324bc66a4c9dc4823e4d52c89aa05711bca072dab59f3e707e96592f6cd0cefb"}
Sep 06 14:16:43.704935 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.704909    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:43.735068 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.735040    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7542-140355587229136_1289ffd6-90c3-4a0d-8c91-3d996d37e55e/test-curl-7542-140355587229136/0.log"
Sep 06 14:16:43.754458 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.754424    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:43.756540 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.756513    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:43.802172 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.802132    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wm62\" (UniqueName: \"kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62\") pod \"1289ffd6-90c3-4a0d-8c91-3d996d37e55e\" (UID: \"1289ffd6-90c3-4a0d-8c91-3d996d37e55e\") "
Sep 06 14:16:43.804265 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.804241    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62" (OuterVolumeSpecName: "kube-api-access-5wm62") pod "1289ffd6-90c3-4a0d-8c91-3d996d37e55e" (UID: "1289ffd6-90c3-4a0d-8c91-3d996d37e55e"). InnerVolumeSpecName "kube-api-access-5wm62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:16:43.870600 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.870517    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:43.870892 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.870878    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:43.870951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.870895    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:43.870990 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.870952    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:43.873739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.873723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:43.879716 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.879687    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:43.903476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.903442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92jd\" (UniqueName: \"kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd\") pod \"test-curl-7542-140355587229136\" (UID: \"5fcd407e-9502-482f-aa9f-1d4c89f698b3\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:43.903646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:43.903560    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wm62\" (UniqueName: \"kubernetes.io/projected/1289ffd6-90c3-4a0d-8c91-3d996d37e55e-kube-api-access-5wm62\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:16:44.004136 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.004097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92jd\" (UniqueName: \"kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd\") pod \"test-curl-7542-140355587229136\" (UID: \"5fcd407e-9502-482f-aa9f-1d4c89f698b3\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:44.011894 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.011859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92jd\" (UniqueName: \"kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd\") pod \"test-curl-7542-140355587229136\" (UID: \"5fcd407e-9502-482f-aa9f-1d4c89f698b3\") " pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:44.184684 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.184648    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:44.452006 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.451975    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:44.453774 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:16:44.453739    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fcd407e_9502_482f_aa9f_1d4c89f698b3.slice/crio-6ebe63aea7374f8e77159eeac3695b998518eb3b7362970e5156edb58b789a32 WatchSource:0}: Error finding container 6ebe63aea7374f8e77159eeac3695b998518eb3b7362970e5156edb58b789a32: Status 404 returned error can't find the container with id 6ebe63aea7374f8e77159eeac3695b998518eb3b7362970e5156edb58b789a32
Sep 06 14:16:44.572361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.572331    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:44.572559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.572369    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df3a0a1af8fdb01ea70a8207ca4cdf985ec5abc115ff5d8d36e4c9d939df3e04"
Sep 06 14:16:44.573557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.573526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229136" event={"ID":"5fcd407e-9502-482f-aa9f-1d4c89f698b3","Type":"ContainerStarted","Data":"6ebe63aea7374f8e77159eeac3695b998518eb3b7362970e5156edb58b789a32"}
Sep 06 14:16:44.583398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:44.583369    2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7542-140355587229136" oldPodUID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3"
Sep 06 14:16:45.580735 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:45.580701    2576 generic.go:356] "Generic (PLEG): container finished" podID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" containerID="2bf47e4686fc584c1a38cd4dcc149f84c2c30e90f6431ace99675c8810753b76" exitCode=0
Sep 06 14:16:45.581241 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:45.580790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229136" event={"ID":"5fcd407e-9502-482f-aa9f-1d4c89f698b3","Type":"ContainerDied","Data":"2bf47e4686fc584c1a38cd4dcc149f84c2c30e90f6431ace99675c8810753b76"}
Sep 06 14:16:45.609614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:45.609582    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1289ffd6-90c3-4a0d-8c91-3d996d37e55e" path="/var/lib/kubelet/pods/1289ffd6-90c3-4a0d-8c91-3d996d37e55e/volumes"
Sep 06 14:16:46.715084 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.715058    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:46.732991 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.732961    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7542-140355587229136_5fcd407e-9502-482f-aa9f-1d4c89f698b3/test-curl-7542-140355587229136/0.log"
Sep 06 14:16:46.758431 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.758286    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:46.760071 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.760048    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7542-140355587229136"]
Sep 06 14:16:46.828855 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.828807    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d92jd\" (UniqueName: \"kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd\") pod \"5fcd407e-9502-482f-aa9f-1d4c89f698b3\" (UID: \"5fcd407e-9502-482f-aa9f-1d4c89f698b3\") "
Sep 06 14:16:46.830900 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.830869    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd" (OuterVolumeSpecName: "kube-api-access-d92jd") pod "5fcd407e-9502-482f-aa9f-1d4c89f698b3" (UID: "5fcd407e-9502-482f-aa9f-1d4c89f698b3"). InnerVolumeSpecName "kube-api-access-d92jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:16:46.929660 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:46.929624    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d92jd\" (UniqueName: \"kubernetes.io/projected/5fcd407e-9502-482f-aa9f-1d4c89f698b3-kube-api-access-d92jd\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:16:47.597677 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:47.597646    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ebe63aea7374f8e77159eeac3695b998518eb3b7362970e5156edb58b789a32"
Sep 06 14:16:47.597677 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:47.597665    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:47.607320 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:16:47.607289    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7542-140355587229136\" is forbidden: User \"system:node:ip-10-0-133-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.ec2.internal' and this object" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" pod="opendatahub/test-curl-7542-140355587229136"
Sep 06 14:16:47.610003 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:47.609980    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" path="/var/lib/kubelet/pods/5fcd407e-9502-482f-aa9f-1d4c89f698b3/volumes"
Sep 06 14:16:50.317088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.317050    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7542-140355587229712"]
Sep 06 14:16:50.317487 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.317364    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:50.317487 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.317376    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:50.317487 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.317438    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fcd407e-9502-482f-aa9f-1d4c89f698b3" containerName="test-curl-7542-140355587229136"
Sep 06 14:16:50.320466 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.320448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:50.322636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.322613    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fdggc\""
Sep 06 14:16:50.325912 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.325887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"]
Sep 06 14:16:50.361020 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.360974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2fh9\" (UniqueName: \"kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9\") pod \"test-curl-7542-140355587229712\" (UID: \"e3c50be7-366e-4a09-9924-b6f4423eed25\") " pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:50.461862 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.461817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2fh9\" (UniqueName: \"kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9\") pod \"test-curl-7542-140355587229712\" (UID: \"e3c50be7-366e-4a09-9924-b6f4423eed25\") " pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:50.469121 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.469091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2fh9\" (UniqueName: \"kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9\") pod \"test-curl-7542-140355587229712\" (UID: \"e3c50be7-366e-4a09-9924-b6f4423eed25\") " pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:50.630520 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.630423    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:50.897832 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:50.897806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"]
Sep 06 14:16:50.899676 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:16:50.899650    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3c50be7_366e_4a09_9924_b6f4423eed25.slice/crio-82ba43109b6634a4700fb9251959f8aa1dca89deaad0770b212dfc462c187b86 WatchSource:0}: Error finding container 82ba43109b6634a4700fb9251959f8aa1dca89deaad0770b212dfc462c187b86: Status 404 returned error can't find the container with id 82ba43109b6634a4700fb9251959f8aa1dca89deaad0770b212dfc462c187b86
Sep 06 14:16:51.617795 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:51.617745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229712" event={"ID":"e3c50be7-366e-4a09-9924-b6f4423eed25","Type":"ContainerStarted","Data":"82ba43109b6634a4700fb9251959f8aa1dca89deaad0770b212dfc462c187b86"}
Sep 06 14:16:52.624470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:52.624434    2576 generic.go:356] "Generic (PLEG): container finished" podID="e3c50be7-366e-4a09-9924-b6f4423eed25" containerID="082b9950ef74b0b1fc2b11f99267c50ef18e7f5b80469b33850be5114e150f9a" exitCode=0
Sep 06 14:16:52.624870 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:52.624482    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7542-140355587229712" event={"ID":"e3c50be7-366e-4a09-9924-b6f4423eed25","Type":"ContainerDied","Data":"082b9950ef74b0b1fc2b11f99267c50ef18e7f5b80469b33850be5114e150f9a"}
Sep 06 14:16:53.761128 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.761103    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:53.777983 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.777951    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7542-140355587229712_e3c50be7-366e-4a09-9924-b6f4423eed25/test-curl-7542-140355587229712/0.log"
Sep 06 14:16:53.802895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.802856    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"]
Sep 06 14:16:53.806271 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.806243    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7542-140355587229712"]
Sep 06 14:16:53.897729 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.897641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2fh9\" (UniqueName: \"kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9\") pod \"e3c50be7-366e-4a09-9924-b6f4423eed25\" (UID: \"e3c50be7-366e-4a09-9924-b6f4423eed25\") "
Sep 06 14:16:53.899665 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.899638    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9" (OuterVolumeSpecName: "kube-api-access-r2fh9") pod "e3c50be7-366e-4a09-9924-b6f4423eed25" (UID: "e3c50be7-366e-4a09-9924-b6f4423eed25"). InnerVolumeSpecName "kube-api-access-r2fh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:16:53.998710 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:53.998666    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2fh9\" (UniqueName: \"kubernetes.io/projected/e3c50be7-366e-4a09-9924-b6f4423eed25-kube-api-access-r2fh9\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:16:54.635698 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:54.635667    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82ba43109b6634a4700fb9251959f8aa1dca89deaad0770b212dfc462c187b86"
Sep 06 14:16:54.635698 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:54.635670    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:54.645579 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:16:54.645542    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7542-140355587229712\" is forbidden: User \"system:node:ip-10-0-133-52.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-52.ec2.internal' and this object" podUID="e3c50be7-366e-4a09-9924-b6f4423eed25" pod="opendatahub/test-curl-7542-140355587229712"
Sep 06 14:16:55.610105 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:16:55.610071    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c50be7-366e-4a09-9924-b6f4423eed25" path="/var/lib/kubelet/pods/e3c50be7-366e-4a09-9924-b6f4423eed25/volumes"
Sep 06 14:17:36.395692 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.395649    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:17:36.396129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.396006    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c50be7-366e-4a09-9924-b6f4423eed25" containerName="test-curl-7542-140355587229712"
Sep 06 14:17:36.396129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.396019    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c50be7-366e-4a09-9924-b6f4423eed25" containerName="test-curl-7542-140355587229712"
Sep 06 14:17:36.396129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.396077    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c50be7-366e-4a09-9924-b6f4423eed25" containerName="test-curl-7542-140355587229712"
Sep 06 14:17:36.399175 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.399152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.402167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.402137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25ae160a\"/\"e2e-lifecycle-model-25ae160a-kserve-self-signed-certs\""
Sep 06 14:17:36.402315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.402169    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25ae160a\"/\"kube-root-ca.crt\""
Sep 06 14:17:36.402315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.402174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25ae160a\"/\"default-dockercfg-s22rx\""
Sep 06 14:17:36.402315 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.402199    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25ae160a\"/\"openshift-service-ca.crt\""
Sep 06 14:17:36.405858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.405833    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:17:36.512497 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.512681 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.512681 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6k5c\" (UniqueName: \"kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.512681 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.512838 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.512838 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.512801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6k5c\" (UniqueName: \"kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614357 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614594    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.614737 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.614653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.616551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.616531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.616865 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.616842    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.622041 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.622010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6k5c\" (UniqueName: \"kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c\") pod \"e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") " pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.710098 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.710057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:36.985376 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:36.985223    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:17:36.988237 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:17:36.988209    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6aa74a9_2cdd_4f01_b48b_cae2628fbb8e.slice/crio-9ae0211b7eb4a7636827946d380888cef03b02d79baa9d2eb822fc2db185b9cf WatchSource:0}: Error finding container 9ae0211b7eb4a7636827946d380888cef03b02d79baa9d2eb822fc2db185b9cf: Status 404 returned error can't find the container with id 9ae0211b7eb4a7636827946d380888cef03b02d79baa9d2eb822fc2db185b9cf
Sep 06 14:17:37.854575 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:37.854538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" event={"ID":"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e","Type":"ContainerStarted","Data":"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"}
Sep 06 14:17:37.855081 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:37.854587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" event={"ID":"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e","Type":"ContainerStarted","Data":"9ae0211b7eb4a7636827946d380888cef03b02d79baa9d2eb822fc2db185b9cf"}
Sep 06 14:17:37.868938 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:37.868888    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" podStartSLOduration=1.8688716589999999 podStartE2EDuration="1.868871659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:17:37.868327942 +0000 UTC m=+1194.894393116" watchObservedRunningTime="2026-09-06 14:17:37.868871659 +0000 UTC m=+1194.894936816"
Sep 06 14:17:43.485439 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:43.485407    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:17:43.577552 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:43.577517    2576 scope.go:117] "RemoveContainer" containerID="2fc5f4b8fd6369079bc52bad771f27dd136d49203146176e6efd44713eacf699"
Sep 06 14:17:43.631654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:43.631624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:17:46.710196 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:46.710155    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:46.710196 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:46.710204    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:46.722955 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:46.722917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:17:46.910212 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:17:46.910178    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:18:01.982029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:01.981988    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"]
Sep 06 14:18:01.986476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:01.986456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:01.988496 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:01.988472    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:18:01.988628 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:01.988474    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:18:01.992577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:01.992547    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"]
Sep 06 14:18:02.064142 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.064096    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z\" (UID: \"656340a1-107b-44cc-aa0e-380a7ca4cf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:02.165118 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.165074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z\" (UID: \"656340a1-107b-44cc-aa0e-380a7ca4cf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:02.165733 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.165711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z\" (UID: \"656340a1-107b-44cc-aa0e-380a7ca4cf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:02.300504 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.300412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:02.571891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.571805    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"]
Sep 06 14:18:02.574850 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:18:02.574824    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod656340a1_107b_44cc_aa0e_380a7ca4cf06.slice/crio-fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be WatchSource:0}: Error finding container fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be: Status 404 returned error can't find the container with id fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be
Sep 06 14:18:02.977965 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.977924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z" event={"ID":"656340a1-107b-44cc-aa0e-380a7ca4cf06","Type":"ContainerStarted","Data":"f3f51ccbe0144f5c945e96caa39779e27c9c1c1e144e1eff6dbdbd16668b9c60"}
Sep 06 14:18:02.978141 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.977984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z" event={"ID":"656340a1-107b-44cc-aa0e-380a7ca4cf06","Type":"ContainerStarted","Data":"fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be"}
Sep 06 14:18:02.992907 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:02.992854    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z" podStartSLOduration=1.992836927 podStartE2EDuration="1.992836927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:18:02.99051098 +0000 UTC m=+1220.016576157" watchObservedRunningTime="2026-09-06 14:18:02.992836927 +0000 UTC m=+1220.018902101"
Sep 06 14:18:03.983928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:03.983891    2576 generic.go:356] "Generic (PLEG): container finished" podID="656340a1-107b-44cc-aa0e-380a7ca4cf06" containerID="f3f51ccbe0144f5c945e96caa39779e27c9c1c1e144e1eff6dbdbd16668b9c60" exitCode=0
Sep 06 14:18:03.984124 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:03.983949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z" event={"ID":"656340a1-107b-44cc-aa0e-380a7ca4cf06","Type":"ContainerDied","Data":"f3f51ccbe0144f5c945e96caa39779e27c9c1c1e144e1eff6dbdbd16668b9c60"}
Sep 06 14:18:05.120506 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.120478    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:05.193229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.193186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca\") pod \"656340a1-107b-44cc-aa0e-380a7ca4cf06\" (UID: \"656340a1-107b-44cc-aa0e-380a7ca4cf06\") "
Sep 06 14:18:05.193576 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.193553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "656340a1-107b-44cc-aa0e-380a7ca4cf06" (UID: "656340a1-107b-44cc-aa0e-380a7ca4cf06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:18:05.294693 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.294561    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/656340a1-107b-44cc-aa0e-380a7ca4cf06-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:05.995922 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.995892    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"
Sep 06 14:18:05.996122 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.995928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z" event={"ID":"656340a1-107b-44cc-aa0e-380a7ca4cf06","Type":"ContainerDied","Data":"fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be"}
Sep 06 14:18:05.996122 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:05.995979    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2050ec26e3ba1cc7b928a00eda13c3f8aa04aa2df6c2c529fd9a76735481be"
Sep 06 14:18:17.562072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:17.562034    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"]
Sep 06 14:18:17.566256 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:17.566221    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25ae160a-2hk9z"]
Sep 06 14:18:17.610347 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:17.610312    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="656340a1-107b-44cc-aa0e-380a7ca4cf06" path="/var/lib/kubelet/pods/656340a1-107b-44cc-aa0e-380a7ca4cf06/volumes"
Sep 06 14:18:25.556897 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:25.556851    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:18:40.214450 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.214358    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:18:40.576542 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.576444    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" podUID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" containerName="main" containerID="cri-o://4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48" gracePeriod=45
Sep 06 14:18:40.821830 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.821805    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:18:40.946495 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946460    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.946722 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946572    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.946722 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.946722 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946686    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6k5c\" (UniqueName: \"kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.946906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946748    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.946906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.946798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir\") pod \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\" (UID: \"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e\") "
Sep 06 14:18:40.948043 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.947835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:40.948306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.948276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home" (OuterVolumeSpecName: "home") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:40.948400 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.948287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:40.953080 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.953040    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm" (OuterVolumeSpecName: "dshm") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:40.953327 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.953298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:18:40.953871 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:40.953847    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c" (OuterVolumeSpecName: "kube-api-access-m6k5c") pod "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" (UID: "d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e"). InnerVolumeSpecName "kube-api-access-m6k5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:18:41.048111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048070    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6k5c\" (UniqueName: \"kubernetes.io/projected/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-kube-api-access-m6k5c\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.048111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048103    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.048111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048114    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.048111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048122    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.048384 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048131    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.048384 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.048139    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:41.170373 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.170340    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" containerID="4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48" exitCode=0
Sep 06 14:18:41.170516 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.170395    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"
Sep 06 14:18:41.170516 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.170428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" event={"ID":"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e","Type":"ContainerDied","Data":"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"}
Sep 06 14:18:41.170516 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.170473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8" event={"ID":"d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e","Type":"ContainerDied","Data":"9ae0211b7eb4a7636827946d380888cef03b02d79baa9d2eb822fc2db185b9cf"}
Sep 06 14:18:41.170516 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.170491    2576 scope.go:117] "RemoveContainer" containerID="4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"
Sep 06 14:18:41.179907 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.179888    2576 scope.go:117] "RemoveContainer" containerID="4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"
Sep 06 14:18:41.180204 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:18:41.180182    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48\": container with ID starting with 4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48 not found: ID does not exist" containerID="4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"
Sep 06 14:18:41.180289 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.180217    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48"} err="failed to get container status \"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48\": rpc error: code = NotFound desc = could not find container \"4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48\": container with ID starting with 4736778d640decf2c2062a820288a6639f1f10ebfa23de6d2f51bd5649d70f48 not found: ID does not exist"
Sep 06 14:18:41.190476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.190417    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:18:41.192404 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.192382    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-25ae160a/e2e-lifecycle-model-25ae160a-kserve-cf4c9787f-67nx8"]
Sep 06 14:18:41.397155 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.397066    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:18:41.609434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:41.609400    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" path="/var/lib/kubelet/pods/d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e/volumes"
Sep 06 14:18:43.598295 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:43.598263    2576 scope.go:117] "RemoveContainer" containerID="c7d0f2859af2a784178617c3a225455d62a31e26016a3906c000dc306a1423a8"
Sep 06 14:18:43.607522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:43.607502    2576 scope.go:117] "RemoveContainer" containerID="2c8821a23ac0a748a80d5295443175425b526755ea1944e6d216ce051eff3b59"
Sep 06 14:18:48.987023 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.986985    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:18:48.987394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987341    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="656340a1-107b-44cc-aa0e-380a7ca4cf06" containerName="revoke-keys"
Sep 06 14:18:48.987394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987353    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="656340a1-107b-44cc-aa0e-380a7ca4cf06" containerName="revoke-keys"
Sep 06 14:18:48.987394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987373    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" containerName="main"
Sep 06 14:18:48.987394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987378    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" containerName="main"
Sep 06 14:18:48.987526 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987439    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6aa74a9-2cdd-4f01-b48b-cae2628fbb8e" containerName="main"
Sep 06 14:18:48.987526 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.987447    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="656340a1-107b-44cc-aa0e-380a7ca4cf06" containerName="revoke-keys"
Sep 06 14:18:48.991890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.991869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:48.994160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:48.994135    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"sub-test-model-0aaff3-kserve-self-signed-certs\""
Sep 06 14:18:49.001031 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.001004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:18:49.125116 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.125116 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.125333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.125333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.125333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.125333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.125313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgpq\" (UniqueName: \"kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgpq\" (UniqueName: \"kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226674 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226745 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.226828 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.226774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.228537 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.228519    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.228815 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.228797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.233161 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.233135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgpq\" (UniqueName: \"kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq\") pod \"sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.302007 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.301901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:49.572816 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:49.572787    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:18:49.575161 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:18:49.575133    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcf732d4_7cd2_48da_b77d_342bf22042f3.slice/crio-2bd18d1d245c6356dcab2dfdfa6b3453379598d84226d22740e122b680cab5ad WatchSource:0}: Error finding container 2bd18d1d245c6356dcab2dfdfa6b3453379598d84226d22740e122b680cab5ad: Status 404 returned error can't find the container with id 2bd18d1d245c6356dcab2dfdfa6b3453379598d84226d22740e122b680cab5ad
Sep 06 14:18:50.216710 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:50.216677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" event={"ID":"dcf732d4-7cd2-48da-b77d-342bf22042f3","Type":"ContainerStarted","Data":"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"}
Sep 06 14:18:50.217133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:50.216724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" event={"ID":"dcf732d4-7cd2-48da-b77d-342bf22042f3","Type":"ContainerStarted","Data":"2bd18d1d245c6356dcab2dfdfa6b3453379598d84226d22740e122b680cab5ad"}
Sep 06 14:18:50.233952 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:50.233894    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" podStartSLOduration=2.233878472 podStartE2EDuration="2.233878472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:18:50.231367395 +0000 UTC m=+1267.257432570" watchObservedRunningTime="2026-09-06 14:18:50.233878472 +0000 UTC m=+1267.259943647"
Sep 06 14:18:55.236869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.236819    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" podUID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" containerName="main" containerID="cri-o://0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae" gracePeriod=45
Sep 06 14:18:55.487494 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.487431    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:18:55.589208 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589150    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm8zj\" (UniqueName: \"kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589277    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589418    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589452    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:55.589654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589499    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm\") pod \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\" (UID: \"4e3d12a3-4756-4a15-8478-3ea08e7a45a7\") "
Sep 06 14:18:55.589654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589537    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home" (OuterVolumeSpecName: "home") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:55.589654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589641    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:55.589901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589883    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:55.589959 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589902    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:55.589959 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.589910    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:55.591562 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.591539    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm" (OuterVolumeSpecName: "dshm") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:55.591562 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.591557    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:18:55.591701 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.591663    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj" (OuterVolumeSpecName: "kube-api-access-vm8zj") pod "4e3d12a3-4756-4a15-8478-3ea08e7a45a7" (UID: "4e3d12a3-4756-4a15-8478-3ea08e7a45a7"). InnerVolumeSpecName "kube-api-access-vm8zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:18:55.691447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.691408    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:55.691447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.691437    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:55.691447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:55.691446    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8zj\" (UniqueName: \"kubernetes.io/projected/4e3d12a3-4756-4a15-8478-3ea08e7a45a7-kube-api-access-vm8zj\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.246339 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.246298    2576 generic.go:356] "Generic (PLEG): container finished" podID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" containerID="0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae" exitCode=0
Sep 06 14:18:56.246857 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.246345    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"
Sep 06 14:18:56.246857 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.246387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" event={"ID":"4e3d12a3-4756-4a15-8478-3ea08e7a45a7","Type":"ContainerDied","Data":"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"}
Sep 06 14:18:56.246857 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.246444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24" event={"ID":"4e3d12a3-4756-4a15-8478-3ea08e7a45a7","Type":"ContainerDied","Data":"5e6777605cba4b026799c4ce2b9b1c1b393960ebc2d5c920624a5dbfa98b37c6"}
Sep 06 14:18:56.246857 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.246468    2576 scope.go:117] "RemoveContainer" containerID="0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"
Sep 06 14:18:56.255136 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.255113    2576 scope.go:117] "RemoveContainer" containerID="0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"
Sep 06 14:18:56.255472 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:18:56.255448    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae\": container with ID starting with 0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae not found: ID does not exist" containerID="0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"
Sep 06 14:18:56.255526 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.255481    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae"} err="failed to get container status \"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae\": rpc error: code = NotFound desc = could not find container \"0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae\": container with ID starting with 0c5257dba5cd9bb53104c552f846e2ca455832e3c3d4bef45c341f1d051730ae not found: ID does not exist"
Sep 06 14:18:56.260636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.260607    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:18:56.262864 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.262844    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/auth-test-model-0aaff3-kserve-6fbf954d95-9gv24"]
Sep 06 14:18:56.418255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.418218    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" podUID="390b4051-fda5-44f9-ab68-6df9f6ae4621" containerName="main" containerID="cri-o://e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4" gracePeriod=45
Sep 06 14:18:56.661775 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.661742    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:18:56.801309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801210    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htkrg\" (UniqueName: \"kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801269    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801288    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir\") pod \"390b4051-fda5-44f9-ab68-6df9f6ae4621\" (UID: \"390b4051-fda5-44f9-ab68-6df9f6ae4621\") "
Sep 06 14:18:56.801661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801560    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home" (OuterVolumeSpecName: "home") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:56.801869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801679    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache" (OuterVolumeSpecName: "model-cache") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:56.801869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801738    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.801869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.801785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:56.803577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.803553    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm" (OuterVolumeSpecName: "dshm") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:18:56.803685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.803609    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:18:56.803685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.803624    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg" (OuterVolumeSpecName: "kube-api-access-htkrg") pod "390b4051-fda5-44f9-ab68-6df9f6ae4621" (UID: "390b4051-fda5-44f9-ab68-6df9f6ae4621"). InnerVolumeSpecName "kube-api-access-htkrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:18:56.902699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.902664    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkrg\" (UniqueName: \"kubernetes.io/projected/390b4051-fda5-44f9-ab68-6df9f6ae4621-kube-api-access-htkrg\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.902699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.902692    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.902699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.902701    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/390b4051-fda5-44f9-ab68-6df9f6ae4621-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.902699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.902711    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:56.902999 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:56.902719    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/390b4051-fda5-44f9-ab68-6df9f6ae4621-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:18:57.259206 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.259174    2576 generic.go:356] "Generic (PLEG): container finished" podID="390b4051-fda5-44f9-ab68-6df9f6ae4621" containerID="e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4" exitCode=0
Sep 06 14:18:57.259611 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.259225    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"
Sep 06 14:18:57.259611 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.259233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" event={"ID":"390b4051-fda5-44f9-ab68-6df9f6ae4621","Type":"ContainerDied","Data":"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"}
Sep 06 14:18:57.259611 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.259260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4" event={"ID":"390b4051-fda5-44f9-ab68-6df9f6ae4621","Type":"ContainerDied","Data":"d0b86b58b75ca41c4ee47b21631d723f0d9104188bbe74b22e2793aa16c10fed"}
Sep 06 14:18:57.259611 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.259278    2576 scope.go:117] "RemoveContainer" containerID="e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"
Sep 06 14:18:57.268573 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.268552    2576 scope.go:117] "RemoveContainer" containerID="e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"
Sep 06 14:18:57.268870 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:18:57.268847    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4\": container with ID starting with e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4 not found: ID does not exist" containerID="e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"
Sep 06 14:18:57.268946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.268883    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4"} err="failed to get container status \"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4\": rpc error: code = NotFound desc = could not find container \"e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4\": container with ID starting with e8799f7dfc81f9e373462215dff89a23aefac3426cbdf63beda0a7d4e4c4d3e4 not found: ID does not exist"
Sep 06 14:18:57.278954 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.278921    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:18:57.282342 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.282321    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/auth-test-model-c443d8-kserve-6b8d98675f-6htt4"]
Sep 06 14:18:57.610308 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.610206    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="390b4051-fda5-44f9-ab68-6df9f6ae4621" path="/var/lib/kubelet/pods/390b4051-fda5-44f9-ab68-6df9f6ae4621/volumes"
Sep 06 14:18:57.610834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:57.610805    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" path="/var/lib/kubelet/pods/4e3d12a3-4756-4a15-8478-3ea08e7a45a7/volumes"
Sep 06 14:18:59.302802 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:59.302726    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:59.303211 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:59.302822    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:18:59.315610 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:18:59.315581    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:19:00.287324 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:00.287292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:19:19.996195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996156    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:19:19.996630 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996559    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="390b4051-fda5-44f9-ab68-6df9f6ae4621" containerName="main"
Sep 06 14:19:19.996630 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996573    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="390b4051-fda5-44f9-ab68-6df9f6ae4621" containerName="main"
Sep 06 14:19:19.996630 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996596    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" containerName="main"
Sep 06 14:19:19.996630 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996602    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" containerName="main"
Sep 06 14:19:19.996803 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996652    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="390b4051-fda5-44f9-ab68-6df9f6ae4621" containerName="main"
Sep 06 14:19:19.996803 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.996662    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e3d12a3-4756-4a15-8478-3ea08e7a45a7" containerName="main"
Sep 06 14:19:19.999425 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:19.999410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.001580 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.001558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"openshift-service-ca.crt\""
Sep 06 14:19:20.002414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.002390    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"kube-root-ca.crt\""
Sep 06 14:19:20.002551 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.002432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"sub-test-model-c443d8-kserve-self-signed-certs\""
Sep 06 14:19:20.002626 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.002545    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"default-dockercfg-55pl5\""
Sep 06 14:19:20.006177 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.006153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:19:20.127253 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127220    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.127445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.127445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmz5m\" (UniqueName: \"kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.127445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.127445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127369    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.127445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.127389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.227994 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.227952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmz5m\" (UniqueName: \"kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.228645 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.228622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.230314 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.230289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.230608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.230588    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.235035 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.235012    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmz5m\" (UniqueName: \"kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m\") pod \"sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.310172 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.310077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:20.581338 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:20.581311    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:19:20.583823 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:19:20.583792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a11755_fb27_4fef_9e05_d37f10b38e08.slice/crio-1058789a7322aa1cc3fff501969d975a7f3ca1d80578ca880c29bde9ed751be2 WatchSource:0}: Error finding container 1058789a7322aa1cc3fff501969d975a7f3ca1d80578ca880c29bde9ed751be2: Status 404 returned error can't find the container with id 1058789a7322aa1cc3fff501969d975a7f3ca1d80578ca880c29bde9ed751be2
Sep 06 14:19:21.385461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:21.385419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" event={"ID":"a7a11755-fb27-4fef-9e05-d37f10b38e08","Type":"ContainerStarted","Data":"b7c1d30f75c5394331305320f9d2f1ce2bcb593db489e6270b48df325eb40663"}
Sep 06 14:19:21.385890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:21.385471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" event={"ID":"a7a11755-fb27-4fef-9e05-d37f10b38e08","Type":"ContainerStarted","Data":"1058789a7322aa1cc3fff501969d975a7f3ca1d80578ca880c29bde9ed751be2"}
Sep 06 14:19:21.402129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:21.402074    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" podStartSLOduration=2.402058433 podStartE2EDuration="2.402058433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:19:21.399712046 +0000 UTC m=+1298.425777223" watchObservedRunningTime="2026-09-06 14:19:21.402058433 +0000 UTC m=+1298.428123609"
Sep 06 14:19:30.311137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:30.311087    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:30.311740 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:30.311149    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:30.324022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:30.323988    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:30.441933 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:30.441900    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:19:43.633625 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:43.633595    2576 scope.go:117] "RemoveContainer" containerID="a8d01a73816e69c2ea0237b88716436367f74226f9385c0eea27b9493a424acf"
Sep 06 14:19:54.992197 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:54.992163    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:19:54.996846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:54.996824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:54.998882 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:54.998860    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-61d831\""
Sep 06 14:19:54.999140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:54.999122    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-61d831-dockercfg-nmjk5\""
Sep 06 14:19:55.005045 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.005012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:19:55.057054 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.057001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpx98\" (UniqueName: \"kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.057263 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.057129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.157894 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.157842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpx98\" (UniqueName: \"kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.158104 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.157920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.158646 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.158618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.166080 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.166045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpx98\" (UniqueName: \"kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98\") pod \"payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.307936 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.307842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:55.579299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:55.579189    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:19:55.582249 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:19:55.582216    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf1f646_1197_4a21_94e8_bdebf5f00234.slice/crio-a8aeadaa1dba151856abfb454c10cd1a7885456fdb2410e5e010253d4c26b60d WatchSource:0}: Error finding container a8aeadaa1dba151856abfb454c10cd1a7885456fdb2410e5e010253d4c26b60d: Status 404 returned error can't find the container with id a8aeadaa1dba151856abfb454c10cd1a7885456fdb2410e5e010253d4c26b60d
Sep 06 14:19:56.565512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:56.565472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" event={"ID":"dcf1f646-1197-4a21-94e8-bdebf5f00234","Type":"ContainerStarted","Data":"a8aeadaa1dba151856abfb454c10cd1a7885456fdb2410e5e010253d4c26b60d"}
Sep 06 14:19:58.577171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:58.577132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" event={"ID":"dcf1f646-1197-4a21-94e8-bdebf5f00234","Type":"ContainerStarted","Data":"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"}
Sep 06 14:19:58.577549 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:58.577226    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:19:58.592129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:19:58.592072    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" podStartSLOduration=2.27156952 podStartE2EDuration="4.592056519s" totalImagesPullingTime="2.320486999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:19:58.590151923 +0000 UTC m=+1335.616217096" watchObservedRunningTime="2026-09-06 14:19:58.592056519 +0000 UTC m=+1335.618121693"
Sep 06 14:20:09.584014 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:09.583930    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:20:14.434220 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:14.434181    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:20:15.586831 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:15.586796    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:20:18.683116 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.683082    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:20:18.686200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.686177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.688409 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.688385    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9dd616-openshift-default-dockercfg-hqv6w\""
Sep 06 14:20:18.700344 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.700310    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:20:18.784573 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m6vm\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784755 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784726    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.784923 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.785055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.784935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886395 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m6vm\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.886929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.886827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.887186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.887157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.887678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.887438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.887835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.887693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.887915 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.887846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.890501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.890451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.891648 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.890651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.895512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.895486    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m6vm\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:18.897853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:18.897800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token\") pod \"e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") " pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:19.000298 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.000194    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:19.276690 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.276660    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:20:19.278825 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:20:19.278787    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa931a1_995f_4c61_9dc5_958ccb909f36.slice/crio-f65ebfe8936f26c54d445134af20b5838e2af3a871e1378c4749c03e7cbf7ea0 WatchSource:0}: Error finding container f65ebfe8936f26c54d445134af20b5838e2af3a871e1378c4749c03e7cbf7ea0: Status 404 returned error can't find the container with id f65ebfe8936f26c54d445134af20b5838e2af3a871e1378c4749c03e7cbf7ea0
Sep 06 14:20:19.281120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.281084    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:19.281192 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.281159    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:19.281238 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.281191    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:19.685133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.685095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerStarted","Data":"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"}
Sep 06 14:20:19.685516 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.685150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerStarted","Data":"f65ebfe8936f26c54d445134af20b5838e2af3a871e1378c4749c03e7cbf7ea0"}
Sep 06 14:20:19.702799 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:19.702732    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podStartSLOduration=1.702716409 podStartE2EDuration="1.702716409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:20:19.701152555 +0000 UTC m=+1356.727217743" watchObservedRunningTime="2026-09-06 14:20:19.702716409 +0000 UTC m=+1356.728781584"
Sep 06 14:20:20.000617 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:20.000517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:20.006110 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:20.006082    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:20.708187 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:20.708146    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:20.709243 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:20.709208    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:23.380860 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.380825    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:20:23.384321 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.384303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.386554 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.386530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9dd616\""
Sep 06 14:20:23.386698 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.386606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9dd616-dockercfg-q9wdl\""
Sep 06 14:20:23.393919 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.393893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:20:23.532413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.532370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.532620 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.532443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkm4x\" (UniqueName: \"kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.634009 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.633914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.634009 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.633975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkm4x\" (UniqueName: \"kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.634699 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.634677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.641222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.641195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkm4x\" (UniqueName: \"kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x\") pod \"payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.695900 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.695867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:23.971377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:23.971353    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:20:23.973276 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:20:23.973251    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2e179e_2902_4527_b5bf_49d20310948d.slice/crio-2af03b54dd32141f7bbf81fad72ee43d3d800990130c271a876ec4dd2904bbf4 WatchSource:0}: Error finding container 2af03b54dd32141f7bbf81fad72ee43d3d800990130c271a876ec4dd2904bbf4: Status 404 returned error can't find the container with id 2af03b54dd32141f7bbf81fad72ee43d3d800990130c271a876ec4dd2904bbf4
Sep 06 14:20:24.730031 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:24.729994    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" event={"ID":"bf2e179e-2902-4527-b5bf-49d20310948d","Type":"ContainerStarted","Data":"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"}
Sep 06 14:20:24.730522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:24.730043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" event={"ID":"bf2e179e-2902-4527-b5bf-49d20310948d","Type":"ContainerStarted","Data":"2af03b54dd32141f7bbf81fad72ee43d3d800990130c271a876ec4dd2904bbf4"}
Sep 06 14:20:24.730522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:24.730123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:24.746098 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:24.746049    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" podStartSLOduration=1.746033766 podStartE2EDuration="1.746033766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:20:24.7439142 +0000 UTC m=+1361.769979375" watchObservedRunningTime="2026-09-06 14:20:24.746033766 +0000 UTC m=+1361.772098941"
Sep 06 14:20:29.453842 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.453793    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" podUID="dcf732d4-7cd2-48da-b77d-342bf22042f3" containerName="main" containerID="cri-o://13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786" gracePeriod=45
Sep 06 14:20:29.706654 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.706594    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:20:29.756667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.756629    2576 generic.go:356] "Generic (PLEG): container finished" podID="dcf732d4-7cd2-48da-b77d-342bf22042f3" containerID="13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786" exitCode=0
Sep 06 14:20:29.756667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.756667    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"
Sep 06 14:20:29.756927 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.756708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" event={"ID":"dcf732d4-7cd2-48da-b77d-342bf22042f3","Type":"ContainerDied","Data":"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"}
Sep 06 14:20:29.756927 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.756750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd" event={"ID":"dcf732d4-7cd2-48da-b77d-342bf22042f3","Type":"ContainerDied","Data":"2bd18d1d245c6356dcab2dfdfa6b3453379598d84226d22740e122b680cab5ad"}
Sep 06 14:20:29.756927 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.756799    2576 scope.go:117] "RemoveContainer" containerID="13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"
Sep 06 14:20:29.765742 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.765722    2576 scope.go:117] "RemoveContainer" containerID="13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"
Sep 06 14:20:29.766076 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:20:29.766055    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786\": container with ID starting with 13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786 not found: ID does not exist" containerID="13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"
Sep 06 14:20:29.766153 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.766090    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786"} err="failed to get container status \"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786\": rpc error: code = NotFound desc = could not find container \"13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786\": container with ID starting with 13fbbd50b591e333742d0a3f6b3a403698cefe6d31a220b524a98de9933b4786 not found: ID does not exist"
Sep 06 14:20:29.783913 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.783882    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784078 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.783938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784078 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784078 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784027    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784257 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784079    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784257 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784113    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbgpq\" (UniqueName: \"kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq\") pod \"dcf732d4-7cd2-48da-b77d-342bf22042f3\" (UID: \"dcf732d4-7cd2-48da-b77d-342bf22042f3\") "
Sep 06 14:20:29.784257 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784154    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home" (OuterVolumeSpecName: "home") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:29.784421 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784328    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:29.784421 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784376    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:29.784525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784475    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:29.784525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784487    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:29.784525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.784497    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:29.786230 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.786200    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm" (OuterVolumeSpecName: "dshm") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:29.786473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.786436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:20:29.786473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.786452    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq" (OuterVolumeSpecName: "kube-api-access-tbgpq") pod "dcf732d4-7cd2-48da-b77d-342bf22042f3" (UID: "dcf732d4-7cd2-48da-b77d-342bf22042f3"). InnerVolumeSpecName "kube-api-access-tbgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:20:29.885728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.885684    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbgpq\" (UniqueName: \"kubernetes.io/projected/dcf732d4-7cd2-48da-b77d-342bf22042f3-kube-api-access-tbgpq\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:29.885728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.885723    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcf732d4-7cd2-48da-b77d-342bf22042f3-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:29.885728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:29.885734    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcf732d4-7cd2-48da-b77d-342bf22042f3-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.077221 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.077190    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:20:30.081022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.080994    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/sub-test-model-0aaff3-kserve-68544d6d5c-4tkxd"]
Sep 06 14:20:30.607199 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.606987    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" podUID="a7a11755-fb27-4fef-9e05-d37f10b38e08" containerName="main" containerID="cri-o://b7c1d30f75c5394331305320f9d2f1ce2bcb593db489e6270b48df325eb40663" gracePeriod=45
Sep 06 14:20:30.764951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.764920    2576 generic.go:356] "Generic (PLEG): container finished" podID="a7a11755-fb27-4fef-9e05-d37f10b38e08" containerID="b7c1d30f75c5394331305320f9d2f1ce2bcb593db489e6270b48df325eb40663" exitCode=0
Sep 06 14:20:30.765131 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.764997    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" event={"ID":"a7a11755-fb27-4fef-9e05-d37f10b38e08","Type":"ContainerDied","Data":"b7c1d30f75c5394331305320f9d2f1ce2bcb593db489e6270b48df325eb40663"}
Sep 06 14:20:30.868858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.868791    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:20:30.897623 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897583    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.897834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897710    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.897834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897746    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.897834 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897818    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmz5m\" (UniqueName: \"kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.897993 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897850    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.897993 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897852    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:30.897993 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:30.898152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.897978    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs\") pod \"a7a11755-fb27-4fef-9e05-d37f10b38e08\" (UID: \"a7a11755-fb27-4fef-9e05-d37f10b38e08\") "
Sep 06 14:20:30.898152 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.898107    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home" (OuterVolumeSpecName: "home") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:30.898379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.898361    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.898453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.898380    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.898453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.898390    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.899948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.899927    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:20:30.900333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.900310    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm" (OuterVolumeSpecName: "dshm") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:20:30.900333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.900324    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m" (OuterVolumeSpecName: "kube-api-access-vmz5m") pod "a7a11755-fb27-4fef-9e05-d37f10b38e08" (UID: "a7a11755-fb27-4fef-9e05-d37f10b38e08"). InnerVolumeSpecName "kube-api-access-vmz5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:20:30.999577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.999534    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmz5m\" (UniqueName: \"kubernetes.io/projected/a7a11755-fb27-4fef-9e05-d37f10b38e08-kube-api-access-vmz5m\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.999577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.999572    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a11755-fb27-4fef-9e05-d37f10b38e08-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:30.999577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:30.999587    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a11755-fb27-4fef-9e05-d37f10b38e08-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:31.609814 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.609778    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf732d4-7cd2-48da-b77d-342bf22042f3" path="/var/lib/kubelet/pods/dcf732d4-7cd2-48da-b77d-342bf22042f3/volumes"
Sep 06 14:20:31.776531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.776495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc" event={"ID":"a7a11755-fb27-4fef-9e05-d37f10b38e08","Type":"ContainerDied","Data":"1058789a7322aa1cc3fff501969d975a7f3ca1d80578ca880c29bde9ed751be2"}
Sep 06 14:20:31.776531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.776539    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"
Sep 06 14:20:31.776790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.776551    2576 scope.go:117] "RemoveContainer" containerID="b7c1d30f75c5394331305320f9d2f1ce2bcb593db489e6270b48df325eb40663"
Sep 06 14:20:31.791944 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.791918    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:20:31.793440 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:31.793418    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/sub-test-model-c443d8-kserve-79cbc5c7f5-j4vjc"]
Sep 06 14:20:33.611537 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:33.611503    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a11755-fb27-4fef-9e05-d37f10b38e08" path="/var/lib/kubelet/pods/a7a11755-fb27-4fef-9e05-d37f10b38e08/volumes"
Sep 06 14:20:35.739528 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.739495    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:20:35.792964 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.792931    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:20:35.793307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793291    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a11755-fb27-4fef-9e05-d37f10b38e08" containerName="main"
Sep 06 14:20:35.793354 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793308    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a11755-fb27-4fef-9e05-d37f10b38e08" containerName="main"
Sep 06 14:20:35.793354 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793331    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf732d4-7cd2-48da-b77d-342bf22042f3" containerName="main"
Sep 06 14:20:35.793354 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793337    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf732d4-7cd2-48da-b77d-342bf22042f3" containerName="main"
Sep 06 14:20:35.793447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793398    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a11755-fb27-4fef-9e05-d37f10b38e08" containerName="main"
Sep 06 14:20:35.793447 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.793409    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf732d4-7cd2-48da-b77d-342bf22042f3" containerName="main"
Sep 06 14:20:35.798171 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.798137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.800325 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.800298    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"rate-test-model-0aaff3-kserve-self-signed-certs\""
Sep 06 14:20:35.800512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.800326    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"openshift-service-ca.crt\""
Sep 06 14:20:35.800512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.800421    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"default-dockercfg-lnc9k\""
Sep 06 14:20:35.800601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.800517    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0aaff3\"/\"kube-root-ca.crt\""
Sep 06 14:20:35.804075 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.804051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:20:35.846794 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.846741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmlx\" (UniqueName: \"kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.846982 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.846872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.846982 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.846924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.846982 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.846968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.847167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.847047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.847167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.847128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmlx\" (UniqueName: \"kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.947908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.947892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.948129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.948108    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.948200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.948160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.948346 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.948319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.950141 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.950115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.950394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.950373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:35.955420 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:35.955395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmlx\" (UniqueName: \"kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx\") pod \"rate-test-model-0aaff3-kserve-5bc784467-fktj7\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") " pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:36.109287 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:36.109184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:36.381378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:36.381300    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:20:36.385917 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:20:36.385878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd94fed4_8cba_4da3_ac7e_19c71e6b444e.slice/crio-8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4 WatchSource:0}: Error finding container 8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4: Status 404 returned error can't find the container with id 8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4
Sep 06 14:20:36.805063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:36.805024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" event={"ID":"fd94fed4-8cba-4da3-ac7e-19c71e6b444e","Type":"ContainerStarted","Data":"a367536f7b890750fde12aa707818962ebf711a86d58ec7f7c4c8565fad1cbb2"}
Sep 06 14:20:36.805063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:36.805067    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" event={"ID":"fd94fed4-8cba-4da3-ac7e-19c71e6b444e","Type":"ContainerStarted","Data":"8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4"}
Sep 06 14:20:36.822639 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:36.822570    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" podStartSLOduration=1.8225475100000001 podStartE2EDuration="1.82254751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:20:36.819450166 +0000 UTC m=+1373.845515341" watchObservedRunningTime="2026-09-06 14:20:36.82254751 +0000 UTC m=+1373.848612685"
Sep 06 14:20:43.661133 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:43.661098    2576 scope.go:117] "RemoveContainer" containerID="848ee019ecd274a97350c3bf7e4057b8464a7582da640cf572e49d6d6f2a49d6"
Sep 06 14:20:43.669506 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:43.669483    2576 scope.go:117] "RemoveContainer" containerID="89578e6a1b97a0604696d94d857166a99c1604d32d12a81331694fe3389f7dd7"
Sep 06 14:20:46.110294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:46.110248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:46.110717 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:46.110325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:46.145364 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:46.145334    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:46.863393 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:46.863358    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:20:49.218345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.218300    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"]
Sep 06 14:20:49.223797 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.223741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:49.226190 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.226157    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:20:49.226451 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.226429    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:20:49.232228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.232190    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"]
Sep 06 14:20:49.281046 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.281008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d831-qbgqq\" (UID: \"15a0abb5-81d0-4cac-8451-960f6012b899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:49.382661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.382616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d831-qbgqq\" (UID: \"15a0abb5-81d0-4cac-8451-960f6012b899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:49.383286 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.383262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d831-qbgqq\" (UID: \"15a0abb5-81d0-4cac-8451-960f6012b899\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:49.543846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.543746    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:49.824590 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.824559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"]
Sep 06 14:20:49.826616 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:20:49.826589    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a0abb5_81d0_4cac_8451_960f6012b899.slice/crio-daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92 WatchSource:0}: Error finding container daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92: Status 404 returned error can't find the container with id daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92
Sep 06 14:20:49.870081 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:49.870041    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq" event={"ID":"15a0abb5-81d0-4cac-8451-960f6012b899","Type":"ContainerStarted","Data":"daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92"}
Sep 06 14:20:50.878876 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:50.878778    2576 generic.go:356] "Generic (PLEG): container finished" podID="15a0abb5-81d0-4cac-8451-960f6012b899" containerID="17356aa1512a3f82ad3b84ff0077afecb81e060ced72a4ba87065c3862a5a822" exitCode=0
Sep 06 14:20:50.878876 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:50.878829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq" event={"ID":"15a0abb5-81d0-4cac-8451-960f6012b899","Type":"ContainerDied","Data":"17356aa1512a3f82ad3b84ff0077afecb81e060ced72a4ba87065c3862a5a822"}
Sep 06 14:20:51.749984 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:51.749927    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 06 14:20:51.750199 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:51.749993    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 06 14:20:52.021777 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.021735    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:52.113553 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.113510    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca\") pod \"15a0abb5-81d0-4cac-8451-960f6012b899\" (UID: \"15a0abb5-81d0-4cac-8451-960f6012b899\") "
Sep 06 14:20:52.113895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.113868    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15a0abb5-81d0-4cac-8451-960f6012b899" (UID: "15a0abb5-81d0-4cac-8451-960f6012b899"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:20:52.215417 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.215374    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15a0abb5-81d0-4cac-8451-960f6012b899-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:20:52.892931 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.892835    2576 generic.go:356] "Generic (PLEG): container finished" podID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerID="b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b" exitCode=0
Sep 06 14:20:52.892931 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.892907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerDied","Data":"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"}
Sep 06 14:20:52.893418 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.893399    2576 scope.go:117] "RemoveContainer" containerID="b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"
Sep 06 14:20:52.894486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq" event={"ID":"15a0abb5-81d0-4cac-8451-960f6012b899","Type":"ContainerDied","Data":"daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92"}
Sep 06 14:20:52.894577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894503    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf5a56819c6255be7dfa4ceb658e2470d93b595db2a8cc3855e9693db506d92"
Sep 06 14:20:52.894577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894475    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"
Sep 06 14:20:52.894656 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894624    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:52.894700 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894685    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:52.894736 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:52.894722    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:20:53.901364 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:53.901322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerStarted","Data":"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"}
Sep 06 14:20:54.001215 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:54.001170    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:54.001549 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:54.001526    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 06 14:20:54.001599 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:54.001577    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 06 14:20:55.001637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:55.001596    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 06 14:20:55.002160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:55.001667    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 06 14:20:56.001404 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:56.001361    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 06 14:20:56.001594 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:56.001425    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 06 14:20:57.001463 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:57.001420    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 06 14:20:57.001933 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:57.001488    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 06 14:20:58.000993 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:58.000953    2576 patch_prober.go:28] interesting pod/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 06 14:20:58.001177 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:58.001012    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 06 14:20:59.004148 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:59.004112    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:59.004528 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:59.004405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:20:59.005086 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:20:59.005066    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:21:04.642756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.642714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:21:04.643355 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.643034    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" podUID="dcf1f646-1197-4a21-94e8-bdebf5f00234" containerName="payload-pre-processing" containerID="cri-o://35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d" gracePeriod=30
Sep 06 14:21:04.780910 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.780865    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"]
Sep 06 14:21:04.783550 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.783520    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d831-qbgqq"]
Sep 06 14:21:04.901984 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.901916    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:21:04.943990 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.943944    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpx98\" (UniqueName: \"kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98\") pod \"dcf1f646-1197-4a21-94e8-bdebf5f00234\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") "
Sep 06 14:21:04.944186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.944001    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume\") pod \"dcf1f646-1197-4a21-94e8-bdebf5f00234\" (UID: \"dcf1f646-1197-4a21-94e8-bdebf5f00234\") "
Sep 06 14:21:04.944425 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.944393    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcf1f646-1197-4a21-94e8-bdebf5f00234" (UID: "dcf1f646-1197-4a21-94e8-bdebf5f00234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:04.946141 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.946113    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98" (OuterVolumeSpecName: "kube-api-access-jpx98") pod "dcf1f646-1197-4a21-94e8-bdebf5f00234" (UID: "dcf1f646-1197-4a21-94e8-bdebf5f00234"). InnerVolumeSpecName "kube-api-access-jpx98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:04.965446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.965409    2576 generic.go:356] "Generic (PLEG): container finished" podID="dcf1f646-1197-4a21-94e8-bdebf5f00234" containerID="35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d" exitCode=0
Sep 06 14:21:04.965613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.965461    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"
Sep 06 14:21:04.965613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.965491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" event={"ID":"dcf1f646-1197-4a21-94e8-bdebf5f00234","Type":"ContainerDied","Data":"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"}
Sep 06 14:21:04.965613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.965549    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk" event={"ID":"dcf1f646-1197-4a21-94e8-bdebf5f00234","Type":"ContainerDied","Data":"a8aeadaa1dba151856abfb454c10cd1a7885456fdb2410e5e010253d4c26b60d"}
Sep 06 14:21:04.965613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.965575    2576 scope.go:117] "RemoveContainer" containerID="35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"
Sep 06 14:21:04.974480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.974452    2576 scope.go:117] "RemoveContainer" containerID="35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"
Sep 06 14:21:04.974796 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:21:04.974753    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d\": container with ID starting with 35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d not found: ID does not exist" containerID="35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"
Sep 06 14:21:04.974855 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.974807    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d"} err="failed to get container status \"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d\": rpc error: code = NotFound desc = could not find container \"35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d\": container with ID starting with 35d34305c23da13c4689702d8ae0ea0aea026bd7af008d28f1f633fd6522163d not found: ID does not exist"
Sep 06 14:21:04.985790 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.985747    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:21:04.989110 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:04.989081    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-61d831-7c55f8775d-dpdzk"]
Sep 06 14:21:05.044890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:05.044848    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpx98\" (UniqueName: \"kubernetes.io/projected/dcf1f646-1197-4a21-94e8-bdebf5f00234-kube-api-access-jpx98\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:05.044890 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:05.044880    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcf1f646-1197-4a21-94e8-bdebf5f00234-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:05.609955 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:05.609925    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15a0abb5-81d0-4cac-8451-960f6012b899" path="/var/lib/kubelet/pods/15a0abb5-81d0-4cac-8451-960f6012b899/volumes"
Sep 06 14:21:05.610309 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:05.610296    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf1f646-1197-4a21-94e8-bdebf5f00234" path="/var/lib/kubelet/pods/dcf1f646-1197-4a21-94e8-bdebf5f00234/volumes"
Sep 06 14:21:11.856351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856313    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"]
Sep 06 14:21:11.856740 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856660    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a0abb5-81d0-4cac-8451-960f6012b899" containerName="revoke-keys"
Sep 06 14:21:11.856740 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856672    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a0abb5-81d0-4cac-8451-960f6012b899" containerName="revoke-keys"
Sep 06 14:21:11.856740 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856683    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf1f646-1197-4a21-94e8-bdebf5f00234" containerName="payload-pre-processing"
Sep 06 14:21:11.856740 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856689    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf1f646-1197-4a21-94e8-bdebf5f00234" containerName="payload-pre-processing"
Sep 06 14:21:11.856920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856757    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf1f646-1197-4a21-94e8-bdebf5f00234" containerName="payload-pre-processing"
Sep 06 14:21:11.856920 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.856788    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a0abb5-81d0-4cac-8451-960f6012b899" containerName="revoke-keys"
Sep 06 14:21:11.861101 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.861081    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:11.863362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.863333    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:21:11.864276 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.864251    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:21:11.865680 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.865650    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"]
Sep 06 14:21:11.913679 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:11.913635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dd616-9m78z\" (UID: \"7432e627-97a4-43a2-9bf0-07cccdf0b037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:12.014684 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:12.014640    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dd616-9m78z\" (UID: \"7432e627-97a4-43a2-9bf0-07cccdf0b037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:12.015294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:12.015271    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9dd616-9m78z\" (UID: \"7432e627-97a4-43a2-9bf0-07cccdf0b037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:12.173945 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:12.173919    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:12.443663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:12.443640    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"]
Sep 06 14:21:12.446283 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:21:12.446249    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7432e627_97a4_43a2_9bf0_07cccdf0b037.slice/crio-4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180 WatchSource:0}: Error finding container 4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180: Status 404 returned error can't find the container with id 4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180
Sep 06 14:21:13.009438 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.009346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z" event={"ID":"7432e627-97a4-43a2-9bf0-07cccdf0b037","Type":"ContainerStarted","Data":"bdb45a00783fc2225d815fabeb5a8de19ff634edbff420b09c55a34ea3e087a3"}
Sep 06 14:21:13.009438 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.009400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z" event={"ID":"7432e627-97a4-43a2-9bf0-07cccdf0b037","Type":"ContainerStarted","Data":"4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180"}
Sep 06 14:21:13.022633 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.022584    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z" podStartSLOduration=2.022569161 podStartE2EDuration="2.022569161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:21:13.021909576 +0000 UTC m=+1410.047974748" watchObservedRunningTime="2026-09-06 14:21:13.022569161 +0000 UTC m=+1410.048634370"
Sep 06 14:21:13.579218 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.579181    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:21:13.582450 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.582433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.585538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.585505    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"openshift-service-ca.crt\""
Sep 06 14:21:13.585538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.585519    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"rate-test-model-c443d8-kserve-self-signed-certs\""
Sep 06 14:21:13.585538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.585533    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"kube-root-ca.crt\""
Sep 06 14:21:13.585825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.585622    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c443d8\"/\"default-dockercfg-55pl5\""
Sep 06 14:21:13.589866 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.589843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:21:13.629365 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.629365 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.629581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.629581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.629581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngm85\" (UniqueName: \"kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.629581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.629538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.730895 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.730850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngm85\" (UniqueName: \"kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731091 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.730929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731091 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.730973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731091 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.730996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.731124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.731176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731392 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.731332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.731430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.731511 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.731475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.733265 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.733242    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.733501 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.733481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.738053 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.738031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngm85\" (UniqueName: \"kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85\") pod \"rate-test-model-c443d8-kserve-7d66bff899-wx62q\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") " pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:13.894060 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:13.893961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:14.019066 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:14.019019    2576 generic.go:356] "Generic (PLEG): container finished" podID="7432e627-97a4-43a2-9bf0-07cccdf0b037" containerID="bdb45a00783fc2225d815fabeb5a8de19ff634edbff420b09c55a34ea3e087a3" exitCode=0
Sep 06 14:21:14.019528 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:14.019079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z" event={"ID":"7432e627-97a4-43a2-9bf0-07cccdf0b037","Type":"ContainerDied","Data":"bdb45a00783fc2225d815fabeb5a8de19ff634edbff420b09c55a34ea3e087a3"}
Sep 06 14:21:14.165651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:14.165617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:21:14.168180 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:21:14.168145    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cf6b5d3_8ae4_4350_aff3_c64adc42e44b.slice/crio-3bea1b5272e71498c54b054b193b5e31f14220ece45075cd61785cb847af2cad WatchSource:0}: Error finding container 3bea1b5272e71498c54b054b193b5e31f14220ece45075cd61785cb847af2cad: Status 404 returned error can't find the container with id 3bea1b5272e71498c54b054b193b5e31f14220ece45075cd61785cb847af2cad
Sep 06 14:21:15.025374 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.025332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" event={"ID":"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b","Type":"ContainerStarted","Data":"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"}
Sep 06 14:21:15.025374 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.025380    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" event={"ID":"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b","Type":"ContainerStarted","Data":"3bea1b5272e71498c54b054b193b5e31f14220ece45075cd61785cb847af2cad"}
Sep 06 14:21:15.042055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.041977    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" podStartSLOduration=2.041952641 podStartE2EDuration="2.041952641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:21:15.040217356 +0000 UTC m=+1412.066282533" watchObservedRunningTime="2026-09-06 14:21:15.041952641 +0000 UTC m=+1412.068017815"
Sep 06 14:21:15.154724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.154691    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:15.245846 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.245802    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca\") pod \"7432e627-97a4-43a2-9bf0-07cccdf0b037\" (UID: \"7432e627-97a4-43a2-9bf0-07cccdf0b037\") "
Sep 06 14:21:15.246184 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.246159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7432e627-97a4-43a2-9bf0-07cccdf0b037" (UID: "7432e627-97a4-43a2-9bf0-07cccdf0b037"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:15.347428 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:15.347336    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7432e627-97a4-43a2-9bf0-07cccdf0b037-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:16.031359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:16.031330    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"
Sep 06 14:21:16.031734 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:16.031364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z" event={"ID":"7432e627-97a4-43a2-9bf0-07cccdf0b037","Type":"ContainerDied","Data":"4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180"}
Sep 06 14:21:16.031734 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:16.031418    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4da9d995ee2919510565283bc8fd562878a01e7f1e8a5aed0178fba2a9bfe180"
Sep 06 14:21:23.894996 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:23.894958    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:23.894996 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:23.895004    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:23.907370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:23.907344    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:24.086727 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:24.086697    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:21:27.345108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.345068    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:21:27.345520 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.345384    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" podUID="bf2e179e-2902-4527-b5bf-49d20310948d" containerName="payload-pre-processing" containerID="cri-o://f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0" gracePeriod=30
Sep 06 14:21:27.478514 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.478469    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"]
Sep 06 14:21:27.481271 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.481245    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9dd616-9m78z"]
Sep 06 14:21:27.610667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.610636    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7432e627-97a4-43a2-9bf0-07cccdf0b037" path="/var/lib/kubelet/pods/7432e627-97a4-43a2-9bf0-07cccdf0b037/volumes"
Sep 06 14:21:27.626586 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.626560    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:21:27.768578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.768540    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume\") pod \"bf2e179e-2902-4527-b5bf-49d20310948d\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") "
Sep 06 14:21:27.768730 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.768609    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkm4x\" (UniqueName: \"kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x\") pod \"bf2e179e-2902-4527-b5bf-49d20310948d\" (UID: \"bf2e179e-2902-4527-b5bf-49d20310948d\") "
Sep 06 14:21:27.769025 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.768994    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf2e179e-2902-4527-b5bf-49d20310948d" (UID: "bf2e179e-2902-4527-b5bf-49d20310948d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:27.770727 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.770706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x" (OuterVolumeSpecName: "kube-api-access-lkm4x") pod "bf2e179e-2902-4527-b5bf-49d20310948d" (UID: "bf2e179e-2902-4527-b5bf-49d20310948d"). InnerVolumeSpecName "kube-api-access-lkm4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:27.869373 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.869281    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2e179e-2902-4527-b5bf-49d20310948d-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:27.869373 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:27.869313    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkm4x\" (UniqueName: \"kubernetes.io/projected/bf2e179e-2902-4527-b5bf-49d20310948d-kube-api-access-lkm4x\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:28.097445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.097410    2576 generic.go:356] "Generic (PLEG): container finished" podID="bf2e179e-2902-4527-b5bf-49d20310948d" containerID="f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0" exitCode=0
Sep 06 14:21:28.097445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.097448    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"
Sep 06 14:21:28.097644 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.097466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" event={"ID":"bf2e179e-2902-4527-b5bf-49d20310948d","Type":"ContainerDied","Data":"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"}
Sep 06 14:21:28.097644 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.097512    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn" event={"ID":"bf2e179e-2902-4527-b5bf-49d20310948d","Type":"ContainerDied","Data":"2af03b54dd32141f7bbf81fad72ee43d3d800990130c271a876ec4dd2904bbf4"}
Sep 06 14:21:28.097644 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.097535    2576 scope.go:117] "RemoveContainer" containerID="f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"
Sep 06 14:21:28.106617 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.106597    2576 scope.go:117] "RemoveContainer" containerID="f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"
Sep 06 14:21:28.106972 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:21:28.106951    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0\": container with ID starting with f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0 not found: ID does not exist" containerID="f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"
Sep 06 14:21:28.107034 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.106982    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0"} err="failed to get container status \"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0\": rpc error: code = NotFound desc = could not find container \"f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0\": container with ID starting with f849510fb718ad307b674bd1dd7ed123fb393528f9c5dcdf11b6adbb9fc7b1a0 not found: ID does not exist"
Sep 06 14:21:28.117577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.117546    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:21:28.125139 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:28.125079    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9dd616-696dd5bd54-4zljn"]
Sep 06 14:21:29.610195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:29.610162    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2e179e-2902-4527-b5bf-49d20310948d" path="/var/lib/kubelet/pods/bf2e179e-2902-4527-b5bf-49d20310948d/volumes"
Sep 06 14:21:33.637635 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.637589    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:21:33.638120 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.637888    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy" containerID="cri-o://dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95" gracePeriod=30
Sep 06 14:21:33.970719 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.970682    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"]
Sep 06 14:21:33.971079 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2e179e-2902-4527-b5bf-49d20310948d" containerName="payload-pre-processing"
Sep 06 14:21:33.971180 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971081    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2e179e-2902-4527-b5bf-49d20310948d" containerName="payload-pre-processing"
Sep 06 14:21:33.971180 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971095    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7432e627-97a4-43a2-9bf0-07cccdf0b037" containerName="revoke-keys"
Sep 06 14:21:33.971180 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971101    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7432e627-97a4-43a2-9bf0-07cccdf0b037" containerName="revoke-keys"
Sep 06 14:21:33.971180 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971177    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2e179e-2902-4527-b5bf-49d20310948d" containerName="payload-pre-processing"
Sep 06 14:21:33.971360 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.971188    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7432e627-97a4-43a2-9bf0-07cccdf0b037" containerName="revoke-keys"
Sep 06 14:21:33.975647 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.975628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:33.978263 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.978242    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:21:33.978669 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.978644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:21:33.980886 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:33.980861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"]
Sep 06 14:21:34.132103 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:34.132068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j\" (UID: \"adb8f073-a8f1-41f1-8652-2738058fefc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:34.233034 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:34.232935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j\" (UID: \"adb8f073-a8f1-41f1-8652-2738058fefc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:34.233566 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:34.233543    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j\" (UID: \"adb8f073-a8f1-41f1-8652-2738058fefc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:34.288853 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:34.288822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:34.564229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:34.564186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"]
Sep 06 14:21:34.566791 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:21:34.566744    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb8f073_a8f1_41f1_8652_2738058fefc6.slice/crio-702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18 WatchSource:0}: Error finding container 702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18: Status 404 returned error can't find the container with id 702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18
Sep 06 14:21:35.137269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:35.137175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j" event={"ID":"adb8f073-a8f1-41f1-8652-2738058fefc6","Type":"ContainerStarted","Data":"c7061c5ff5a7773f064f4841a7e8bb6a4f6c63409496ed50bf420fc8ee99bb72"}
Sep 06 14:21:35.137269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:35.137234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j" event={"ID":"adb8f073-a8f1-41f1-8652-2738058fefc6","Type":"ContainerStarted","Data":"702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18"}
Sep 06 14:21:35.152112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:35.152054    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j" podStartSLOduration=2.15203598 podStartE2EDuration="2.15203598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:21:35.149698207 +0000 UTC m=+1432.175763383" watchObservedRunningTime="2026-09-06 14:21:35.15203598 +0000 UTC m=+1432.178101155"
Sep 06 14:21:36.143543 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:36.143463    2576 generic.go:356] "Generic (PLEG): container finished" podID="adb8f073-a8f1-41f1-8652-2738058fefc6" containerID="c7061c5ff5a7773f064f4841a7e8bb6a4f6c63409496ed50bf420fc8ee99bb72" exitCode=0
Sep 06 14:21:36.144018 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:36.143547    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j" event={"ID":"adb8f073-a8f1-41f1-8652-2738058fefc6","Type":"ContainerDied","Data":"c7061c5ff5a7773f064f4841a7e8bb6a4f6c63409496ed50bf420fc8ee99bb72"}
Sep 06 14:21:37.289578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:37.289553    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:37.362093 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:37.362048    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca\") pod \"adb8f073-a8f1-41f1-8652-2738058fefc6\" (UID: \"adb8f073-a8f1-41f1-8652-2738058fefc6\") "
Sep 06 14:21:37.362419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:37.362395    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adb8f073-a8f1-41f1-8652-2738058fefc6" (UID: "adb8f073-a8f1-41f1-8652-2738058fefc6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:37.463232 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:37.463197    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb8f073-a8f1-41f1-8652-2738058fefc6-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:38.160809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.160751    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"
Sep 06 14:21:38.160992 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.160785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j" event={"ID":"adb8f073-a8f1-41f1-8652-2738058fefc6","Type":"ContainerDied","Data":"702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18"}
Sep 06 14:21:38.160992 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.160898    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="702a16859d5d36de1d1cb70cab3061a8d3ea20d84d3407692af0639735056f18"
Sep 06 14:21:38.885626 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.885600    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:21:38.973904 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.973849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.973904 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.973909    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974174 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.973940    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974174 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974174 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974140    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974174 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974406 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974169    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:38.974406 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974202    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m6vm\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974406 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974371    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket\") pod \"dfa931a1-995f-4c61-9dc5-958ccb909f36\" (UID: \"dfa931a1-995f-4c61-9dc5-958ccb909f36\") "
Sep 06 14:21:38.974560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974414    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:38.974560 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974456    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:38.974717 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:38.974806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974732    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:38.974806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974753    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa931a1-995f-4c61-9dc5-958ccb909f36-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:38.974806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974796    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:38.974975 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974810    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:38.974975 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.974792    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:38.976269 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.976245    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:21:38.976487 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.976462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm" (OuterVolumeSpecName: "kube-api-access-9m6vm") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "kube-api-access-9m6vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:38.976585 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.976547    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:38.976585 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:38.976563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfa931a1-995f-4c61-9dc5-958ccb909f36" (UID: "dfa931a1-995f-4c61-9dc5-958ccb909f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:39.075476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.075422    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:39.075476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.075468    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:39.075476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.075478    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:39.075476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.075486    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:39.075476 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.075494    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m6vm\" (UniqueName: \"kubernetes.io/projected/dfa931a1-995f-4c61-9dc5-958ccb909f36-kube-api-access-9m6vm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:39.167176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.167141    2576 generic.go:356] "Generic (PLEG): container finished" podID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerID="dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95" exitCode=0
Sep 06 14:21:39.167377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.167203    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"
Sep 06 14:21:39.167377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.167219    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerDied","Data":"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"}
Sep 06 14:21:39.167377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.167273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9" event={"ID":"dfa931a1-995f-4c61-9dc5-958ccb909f36","Type":"ContainerDied","Data":"f65ebfe8936f26c54d445134af20b5838e2af3a871e1378c4749c03e7cbf7ea0"}
Sep 06 14:21:39.167377 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.167295    2576 scope.go:117] "RemoveContainer" containerID="dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"
Sep 06 14:21:39.176498 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.176480    2576 scope.go:117] "RemoveContainer" containerID="b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"
Sep 06 14:21:39.184414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.184397    2576 scope.go:117] "RemoveContainer" containerID="dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"
Sep 06 14:21:39.184679 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:21:39.184660    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95\": container with ID starting with dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95 not found: ID does not exist" containerID="dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"
Sep 06 14:21:39.184748 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.184686    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95"} err="failed to get container status \"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95\": rpc error: code = NotFound desc = could not find container \"dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95\": container with ID starting with dbdd3e62e821afb765d43302bb0bad22ee25aa1f16893d0da05aeb4f0dcd1e95 not found: ID does not exist"
Sep 06 14:21:39.184748 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.184702    2576 scope.go:117] "RemoveContainer" containerID="b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"
Sep 06 14:21:39.184959 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:21:39.184943    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b\": container with ID starting with b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b not found: ID does not exist" containerID="b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"
Sep 06 14:21:39.185001 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.184965    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b"} err="failed to get container status \"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b\": rpc error: code = NotFound desc = could not find container \"b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b\": container with ID starting with b309114dceee3e1aa461fd782b67e577c47fba80693ba96528dfe8e6e6b6d38b not found: ID does not exist"
Sep 06 14:21:39.188008 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.187986    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:21:39.191412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.191389    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9dd616-openshift-default-66668f94db-kjkf9"]
Sep 06 14:21:39.610156 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:39.610119    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" path="/var/lib/kubelet/pods/dfa931a1-995f-4c61-9dc5-958ccb909f36/volumes"
Sep 06 14:21:43.697994 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:43.697960    2576 scope.go:117] "RemoveContainer" containerID="96eda92c8a933fab02cb3be0bb6025fb54f2248bb93108d3805841c8b8fc3830"
Sep 06 14:21:44.778966 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:44.778918    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"]
Sep 06 14:21:44.780803 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:44.780778    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6328f9-wr94j"]
Sep 06 14:21:45.609850 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:45.609813    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb8f073-a8f1-41f1-8652-2738058fefc6" path="/var/lib/kubelet/pods/adb8f073-a8f1-41f1-8652-2738058fefc6/volumes"
Sep 06 14:21:51.036914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:51.036875    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:21:51.037412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:51.037101    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy" containerID="cri-o://1ffadb8ebfd5c5ad8717f80645894e76738a91469ec56a7d962b5c559700bc19" gracePeriod=30
Sep 06 14:21:56.253954 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.253916    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerID="1ffadb8ebfd5c5ad8717f80645894e76738a91469ec56a7d962b5c559700bc19" exitCode=0
Sep 06 14:21:56.254345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.253985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerDied","Data":"1ffadb8ebfd5c5ad8717f80645894e76738a91469ec56a7d962b5c559700bc19"}
Sep 06 14:21:56.254345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.254048    2576 scope.go:117] "RemoveContainer" containerID="7b3fdeb937ba916bbc1b7880501b93c8747574a138212e3f85ff84fa5284f226"
Sep 06 14:21:56.294731 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.294706    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:21:56.341686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.341686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.341686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341666    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341692    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341718    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341739    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341789    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341860    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvtvm\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm\") pod \"5a86078c-c84e-4ddc-8eba-3b03dd640531\" (UID: \"5a86078c-c84e-4ddc-8eba-3b03dd640531\") "
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:56.342004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.341967    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342062    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342227    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342245    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342258    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:21:56.342401 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.342397    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:56.344138 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.344104    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:56.344270 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.344141    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:21:56.344349 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.344332    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:21:56.344419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.344406    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm" (OuterVolumeSpecName: "kube-api-access-mvtvm") pod "5a86078c-c84e-4ddc-8eba-3b03dd640531" (UID: "5a86078c-c84e-4ddc-8eba-3b03dd640531"). InnerVolumeSpecName "kube-api-access-mvtvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:21:56.443414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443369    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.443414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443407    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.443414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443416    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a86078c-c84e-4ddc-8eba-3b03dd640531-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.443414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443423    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.443682 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443431    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a86078c-c84e-4ddc-8eba-3b03dd640531-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:56.443682 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:56.443438    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtvm\" (UniqueName: \"kubernetes.io/projected/5a86078c-c84e-4ddc-8eba-3b03dd640531-kube-api-access-mvtvm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:21:57.260850 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.260814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9" event={"ID":"5a86078c-c84e-4ddc-8eba-3b03dd640531","Type":"ContainerDied","Data":"1622f23e3c741f88939359e8420e96ca301eb8615b934f8065bfff2228a52318"}
Sep 06 14:21:57.261267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.260876    2576 scope.go:117] "RemoveContainer" containerID="1ffadb8ebfd5c5ad8717f80645894e76738a91469ec56a7d962b5c559700bc19"
Sep 06 14:21:57.261267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.260894    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"
Sep 06 14:21:57.281622 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.281591    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:21:57.285132 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.285107    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6328f9-openshift-default-6ddf769678-v9sm9"]
Sep 06 14:21:57.610364 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:21:57.610276    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" path="/var/lib/kubelet/pods/5a86078c-c84e-4ddc-8eba-3b03dd640531/volumes"
Sep 06 14:22:08.469965 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:08.469927    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:22:08.470564 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:08.470161    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" containerID="cri-o://a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db" gracePeriod=30
Sep 06 14:22:10.869564 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:10.869521    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 14:22:10.869975 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:10.869579    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 14:22:13.720191 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.720161    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:22:13.906155 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906065    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906155 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906151    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906239    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906275    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906309    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906346    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppldm\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906375    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:13.906810 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy\") pod \"0d66be43-19c7-4a6b-be6a-26870e65cc43\" (UID: \"0d66be43-19c7-4a6b-be6a-26870e65cc43\") "
Sep 06 14:22:13.906810 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:13.906810 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906577    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:13.906810 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:13.906810 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906725    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:22:13.907056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906879    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:13.907056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906901    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:13.907056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906916    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:13.907056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906929    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:13.907056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.906941    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d66be43-19c7-4a6b-be6a-26870e65cc43-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:13.908525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.908502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:22:13.908525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.908512    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:22:13.908990 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.908968    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:13.909101 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:13.908996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm" (OuterVolumeSpecName: "kube-api-access-ppldm") pod "0d66be43-19c7-4a6b-be6a-26870e65cc43" (UID: "0d66be43-19c7-4a6b-be6a-26870e65cc43"). InnerVolumeSpecName "kube-api-access-ppldm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:22:14.007412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.007371    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:14.007412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.007403    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:14.007412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.007412    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppldm\" (UniqueName: \"kubernetes.io/projected/0d66be43-19c7-4a6b-be6a-26870e65cc43-kube-api-access-ppldm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:14.007412 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.007422    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d66be43-19c7-4a6b-be6a-26870e65cc43-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:14.349083 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.349045    2576 generic.go:356] "Generic (PLEG): container finished" podID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerID="a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db" exitCode=0
Sep 06 14:22:14.349229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.349102    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"
Sep 06 14:22:14.349229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.349140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerDied","Data":"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"}
Sep 06 14:22:14.349229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.349211    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4" event={"ID":"0d66be43-19c7-4a6b-be6a-26870e65cc43","Type":"ContainerDied","Data":"62396bbb5dbe46c50e123ec835023e6b4958d80de2e481a98d761ba3dafadbed"}
Sep 06 14:22:14.349378 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.349245    2576 scope.go:117] "RemoveContainer" containerID="a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"
Sep 06 14:22:14.358451 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.358433    2576 scope.go:117] "RemoveContainer" containerID="cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"
Sep 06 14:22:14.366352 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.366334    2576 scope.go:117] "RemoveContainer" containerID="a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"
Sep 06 14:22:14.366634 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:22:14.366615    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db\": container with ID starting with a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db not found: ID does not exist" containerID="a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"
Sep 06 14:22:14.366683 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.366644    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db"} err="failed to get container status \"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db\": rpc error: code = NotFound desc = could not find container \"a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db\": container with ID starting with a6627927e1af5816a30743faf3335106c723e40c43a74d5b6f71444f568ee1db not found: ID does not exist"
Sep 06 14:22:14.366683 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.366662    2576 scope.go:117] "RemoveContainer" containerID="cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"
Sep 06 14:22:14.366939 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:22:14.366920    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070\": container with ID starting with cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070 not found: ID does not exist" containerID="cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"
Sep 06 14:22:14.366990 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.366945    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070"} err="failed to get container status \"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070\": rpc error: code = NotFound desc = could not find container \"cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070\": container with ID starting with cad3d596a2796dfed36a6517aa052d1f725a28fa3838a08c2bcd45e4aadb1070 not found: ID does not exist"
Sep 06 14:22:14.369504 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.369482    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:22:14.373146 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:14.373123    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a388e1-openshift-default-6bf75c56d4-d9zr4"]
Sep 06 14:22:15.610021 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:15.609987    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" path="/var/lib/kubelet/pods/0d66be43-19c7-4a6b-be6a-26870e65cc43/volumes"
Sep 06 14:22:17.179097 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:17.179054    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:22:18.171994 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:18.171958    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:22:23.699557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699514    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699831    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699843    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699854    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699860    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699873    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699880    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb8f073-a8f1-41f1-8652-2738058fefc6" containerName="revoke-keys"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699884    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb8f073-a8f1-41f1-8652-2738058fefc6" containerName="revoke-keys"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699898    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699903    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699919    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699925    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.699940 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699930    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699980    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699988    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.699995    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.700001    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa931a1-995f-4c61-9dc5-958ccb909f36" containerName="istio-proxy"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.700006    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb8f073-a8f1-41f1-8652-2738058fefc6" containerName="revoke-keys"
Sep 06 14:22:23.700379 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.700012    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d66be43-19c7-4a6b-be6a-26870e65cc43" containerName="istio-proxy"
Sep 06 14:22:23.704466 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.704268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.706636 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.706611    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d46936-dockercfg-xcgv7\""
Sep 06 14:22:23.706749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.706611    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d46936\""
Sep 06 14:22:23.713101 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.713071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:22:23.716016 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.715990    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:22:23.716541 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.716521    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a86078c-c84e-4ddc-8eba-3b03dd640531" containerName="istio-proxy"
Sep 06 14:22:23.719774 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.719739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.728602 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.728561    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:22:23.786728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.786692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrw7\" (UniqueName: \"kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.786728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.786729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.787218 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.786815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.787218 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.786837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9k8\" (UniqueName: \"kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.888108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrw7\" (UniqueName: \"kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.888108 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.888386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.888386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888182    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9k8\" (UniqueName: \"kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.888948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:23.889015 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.888935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.896267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.896234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9k8\" (UniqueName: \"kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8\") pod \"payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:23.896267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:23.896237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrw7\" (UniqueName: \"kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7\") pod \"payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:24.015710 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.015619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:24.030912 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.030885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:24.293229 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.293203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:22:24.295297 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:22:24.295267    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod553bb504_008c_4c47_aa06_3b2e8e46ba4a.slice/crio-61d263b04c4f91233cd134eac8cba8f76e95018a2cf7f1a9ffb5058737b5d02d WatchSource:0}: Error finding container 61d263b04c4f91233cd134eac8cba8f76e95018a2cf7f1a9ffb5058737b5d02d: Status 404 returned error can't find the container with id 61d263b04c4f91233cd134eac8cba8f76e95018a2cf7f1a9ffb5058737b5d02d
Sep 06 14:22:24.317205 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.317183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:22:24.327949 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:22:24.327913    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03d0596_8350_4430_9c2e_c48e2a5daadf.slice/crio-04c6bb5d395eb78ffefc5da0fee65c5643a01cc2b9b673f398fca5feb9709025 WatchSource:0}: Error finding container 04c6bb5d395eb78ffefc5da0fee65c5643a01cc2b9b673f398fca5feb9709025: Status 404 returned error can't find the container with id 04c6bb5d395eb78ffefc5da0fee65c5643a01cc2b9b673f398fca5feb9709025
Sep 06 14:22:24.400835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.400802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" event={"ID":"553bb504-008c-4c47-aa06-3b2e8e46ba4a","Type":"ContainerStarted","Data":"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"}
Sep 06 14:22:24.401003 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.400858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" event={"ID":"553bb504-008c-4c47-aa06-3b2e8e46ba4a","Type":"ContainerStarted","Data":"61d263b04c4f91233cd134eac8cba8f76e95018a2cf7f1a9ffb5058737b5d02d"}
Sep 06 14:22:24.401003 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.400939    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:24.402531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.402505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" event={"ID":"f03d0596-8350-4430-9c2e-c48e2a5daadf","Type":"ContainerStarted","Data":"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"}
Sep 06 14:22:24.402632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.402555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" event={"ID":"f03d0596-8350-4430-9c2e-c48e2a5daadf","Type":"ContainerStarted","Data":"04c6bb5d395eb78ffefc5da0fee65c5643a01cc2b9b673f398fca5feb9709025"}
Sep 06 14:22:24.402632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.402585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:24.415280 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.415230    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" podStartSLOduration=1.415212187 podStartE2EDuration="1.415212187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:24.414175052 +0000 UTC m=+1481.440240228" watchObservedRunningTime="2026-09-06 14:22:24.415212187 +0000 UTC m=+1481.441277365"
Sep 06 14:22:24.429606 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:24.429561    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" podStartSLOduration=1.429546483 podStartE2EDuration="1.429546483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:24.428602619 +0000 UTC m=+1481.454667796" watchObservedRunningTime="2026-09-06 14:22:24.429546483 +0000 UTC m=+1481.455611658"
Sep 06 14:22:32.201071 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.201030    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" podUID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" containerName="main" containerID="cri-o://a367536f7b890750fde12aa707818962ebf711a86d58ec7f7c4c8565fad1cbb2" gracePeriod=45
Sep 06 14:22:32.441792 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.441742    2576 generic.go:356] "Generic (PLEG): container finished" podID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" containerID="a367536f7b890750fde12aa707818962ebf711a86d58ec7f7c4c8565fad1cbb2" exitCode=0
Sep 06 14:22:32.441908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.441802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" event={"ID":"fd94fed4-8cba-4da3-ac7e-19c71e6b444e","Type":"ContainerDied","Data":"a367536f7b890750fde12aa707818962ebf711a86d58ec7f7c4c8565fad1cbb2"}
Sep 06 14:22:32.441908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.441853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7" event={"ID":"fd94fed4-8cba-4da3-ac7e-19c71e6b444e","Type":"ContainerDied","Data":"8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4"}
Sep 06 14:22:32.441908 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.441865    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ef4d0766d2520f05479af50a729c7c2ecf5f993d8b6fd5416af026cc96acdd4"
Sep 06 14:22:32.448493 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.448469    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:22:32.464157 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.464310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.464310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464234    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.464310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464283    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.464574 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464548    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home" (OuterVolumeSpecName: "home") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:32.464754 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.464721    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:32.466471 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.466439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm" (OuterVolumeSpecName: "dshm") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:32.466729 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.466702    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:22:32.565170 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565119    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.565350 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565256    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msmlx\" (UniqueName: \"kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx\") pod \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\" (UID: \"fd94fed4-8cba-4da3-ac7e-19c71e6b444e\") "
Sep 06 14:22:32.565477 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:32.565530 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565517    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:32.565567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565532    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:32.565567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565541    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:32.565567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.565549    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:32.567334 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.567311    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx" (OuterVolumeSpecName: "kube-api-access-msmlx") pod "fd94fed4-8cba-4da3-ac7e-19c71e6b444e" (UID: "fd94fed4-8cba-4da3-ac7e-19c71e6b444e"). InnerVolumeSpecName "kube-api-access-msmlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:22:32.666716 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.666681    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:32.666716 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:32.666710    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmlx\" (UniqueName: \"kubernetes.io/projected/fd94fed4-8cba-4da3-ac7e-19c71e6b444e-kube-api-access-msmlx\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.192027 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.191992    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" podUID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" containerName="main" containerID="cri-o://a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6" gracePeriod=45
Sep 06 14:22:33.439428 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.439404    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:22:33.447071 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.446999    2576 generic.go:356] "Generic (PLEG): container finished" podID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" containerID="a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6" exitCode=0
Sep 06 14:22:33.447071 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.447038    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"
Sep 06 14:22:33.447228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.447078    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" event={"ID":"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b","Type":"ContainerDied","Data":"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"}
Sep 06 14:22:33.447228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.447131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q" event={"ID":"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b","Type":"ContainerDied","Data":"3bea1b5272e71498c54b054b193b5e31f14220ece45075cd61785cb847af2cad"}
Sep 06 14:22:33.447228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.447150    2576 scope.go:117] "RemoveContainer" containerID="a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"
Sep 06 14:22:33.447228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.447174    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"
Sep 06 14:22:33.458809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.458498    2576 scope.go:117] "RemoveContainer" containerID="a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"
Sep 06 14:22:33.461159 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:22:33.459432    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6\": container with ID starting with a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6 not found: ID does not exist" containerID="a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"
Sep 06 14:22:33.461159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.459481    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6"} err="failed to get container status \"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6\": rpc error: code = NotFound desc = could not find container \"a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6\": container with ID starting with a9afe4ee4b49c1e8b50da0aaa1554caeedc6de7808e6cd64ea6c370731f7d4b6 not found: ID does not exist"
Sep 06 14:22:33.473652 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473625    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.473858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473683    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.473858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473728    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.473858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngm85\" (UniqueName: \"kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.473858 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473835    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.474063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm\") pod \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\" (UID: \"9cf6b5d3-8ae4-4350-aff3-c64adc42e44b\") "
Sep 06 14:22:33.474063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.473943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:33.474063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.474027    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home" (OuterVolumeSpecName: "home") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:33.474063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.474044    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:33.474268 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.474201    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.474268 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.474218    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.474268 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.474231    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.476929 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.476881    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85" (OuterVolumeSpecName: "kube-api-access-ngm85") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "kube-api-access-ngm85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:22:33.477053 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.476998    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:22:33.477240 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.477224    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm" (OuterVolumeSpecName: "dshm") pod "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" (UID: "9cf6b5d3-8ae4-4350-aff3-c64adc42e44b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:22:33.478520 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.478495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:22:33.480291 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.480267    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0aaff3/rate-test-model-0aaff3-kserve-5bc784467-fktj7"]
Sep 06 14:22:33.575547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.575493    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngm85\" (UniqueName: \"kubernetes.io/projected/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-kube-api-access-ngm85\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.575547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.575538    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.575547 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.575554    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:22:33.609970 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.609933    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" path="/var/lib/kubelet/pods/fd94fed4-8cba-4da3-ac7e-19c71e6b444e/volumes"
Sep 06 14:22:33.761274 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.761193    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:22:33.766414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:33.766386    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c443d8/rate-test-model-c443d8-kserve-7d66bff899-wx62q"]
Sep 06 14:22:35.409358 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:35.409329    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:22:35.409842 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:35.409400    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:22:35.609711 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:35.609680    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" path="/var/lib/kubelet/pods/9cf6b5d3-8ae4-4350-aff3-c64adc42e44b/volumes"
Sep 06 14:22:43.513112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:43.513085    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:22:43.658419 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:43.658391    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:22:43.730847 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:43.730816    2576 scope.go:117] "RemoveContainer" containerID="324bc66a4c9dc4823e4d52c89aa05711bca072dab59f3e707e96592f6cd0cefb"
Sep 06 14:22:47.822348 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822308    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:22:47.822825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822807    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" containerName="main"
Sep 06 14:22:47.822885 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822828    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" containerName="main"
Sep 06 14:22:47.822885 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822848    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" containerName="main"
Sep 06 14:22:47.822885 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822857    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" containerName="main"
Sep 06 14:22:47.822986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822934    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd94fed4-8cba-4da3-ac7e-19c71e6b444e" containerName="main"
Sep 06 14:22:47.822986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.822951    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf6b5d3-8ae4-4350-aff3-c64adc42e44b" containerName="main"
Sep 06 14:22:47.826266 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.826247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.829333 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.829310    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c1a83e-openshift-default-dockercfg-psnxz\""
Sep 06 14:22:47.837160 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.837128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:22:47.908999 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.908955    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.908999 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.908997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909259 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qvr\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909533 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:47.909533 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:47.909303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010345 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qvr\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010783 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010843 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010843 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.010953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.010827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.011056 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.011032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.011172 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.011140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.011356 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.011246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.011801 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.011778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.012954 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.012933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.013083 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.013067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.018747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.018726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.018948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.018832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qvr\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr\") pod \"e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") " pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.139366 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.139261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:48.414856 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.414822    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:22:48.417475 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:22:48.417434    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22292ba_6cdb_4d4f_8074_ca6384537ab0.slice/crio-185f12e1b0608179c39b0474b18c926b555bc89d43e05c842131f54cd598482f WatchSource:0}: Error finding container 185f12e1b0608179c39b0474b18c926b555bc89d43e05c842131f54cd598482f: Status 404 returned error can't find the container with id 185f12e1b0608179c39b0474b18c926b555bc89d43e05c842131f54cd598482f
Sep 06 14:22:48.420119 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.420076    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:22:48.420207 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.420166    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:22:48.420246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.420211    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:22:48.525163 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.525121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerStarted","Data":"023976c3d8c2a60c02739f5e5e17d4bae995ebfa1079c03b4f216c0ba57e9bd5"}
Sep 06 14:22:48.525473 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.525190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerStarted","Data":"185f12e1b0608179c39b0474b18c926b555bc89d43e05c842131f54cd598482f"}
Sep 06 14:22:48.542983 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:48.542924    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podStartSLOduration=1.5429068670000001 podStartE2EDuration="1.542906867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:48.540864856 +0000 UTC m=+1505.566930032" watchObservedRunningTime="2026-09-06 14:22:48.542906867 +0000 UTC m=+1505.568972115"
Sep 06 14:22:49.139467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:49.139418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:49.144919 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:49.144889    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:49.535773 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:49.535730    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:49.536611 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:49.536585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:22:52.136170 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.136135    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:22:52.139517 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.139499    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.141906 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.141883    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c1a83e\""
Sep 06 14:22:52.142031 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.142012    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c1a83e-dockercfg-5twrt\""
Sep 06 14:22:52.150714 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.150691    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:22:52.252569 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.252527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmnwn\" (UniqueName: \"kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.252818 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.252587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.354012 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.353968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.354188 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.354094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmnwn\" (UniqueName: \"kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.354633 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.354613    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.361263 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.361236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmnwn\" (UniqueName: \"kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn\") pod \"payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.451710 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.451670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:52.735082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:52.734998    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:22:52.738152 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:22:52.738127    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588b0d32_5d88_4c87_9fbf_ba6ac65b5378.slice/crio-1a7df2babc8340b6f71997e57c0cd6ea3b36a39c5e53bdb107cd336094a7903c WatchSource:0}: Error finding container 1a7df2babc8340b6f71997e57c0cd6ea3b36a39c5e53bdb107cd336094a7903c: Status 404 returned error can't find the container with id 1a7df2babc8340b6f71997e57c0cd6ea3b36a39c5e53bdb107cd336094a7903c
Sep 06 14:22:53.557609 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:53.557575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" event={"ID":"588b0d32-5d88-4c87-9fbf-ba6ac65b5378","Type":"ContainerStarted","Data":"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"}
Sep 06 14:22:53.558023 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:53.557630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" event={"ID":"588b0d32-5d88-4c87-9fbf-ba6ac65b5378","Type":"ContainerStarted","Data":"1a7df2babc8340b6f71997e57c0cd6ea3b36a39c5e53bdb107cd336094a7903c"}
Sep 06 14:22:53.558023 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:53.557707    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:22:53.573022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:22:53.572965    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" podStartSLOduration=1.572947898 podStartE2EDuration="1.572947898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:22:53.570875153 +0000 UTC m=+1510.596940362" watchObservedRunningTime="2026-09-06 14:22:53.572947898 +0000 UTC m=+1510.599013073"
Sep 06 14:23:04.563719 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:04.563683    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:23:09.641381 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.641344    2576 generic.go:356] "Generic (PLEG): container finished" podID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerID="023976c3d8c2a60c02739f5e5e17d4bae995ebfa1079c03b4f216c0ba57e9bd5" exitCode=0
Sep 06 14:23:09.641813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.641399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerDied","Data":"023976c3d8c2a60c02739f5e5e17d4bae995ebfa1079c03b4f216c0ba57e9bd5"}
Sep 06 14:23:09.641813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.641755    2576 scope.go:117] "RemoveContainer" containerID="023976c3d8c2a60c02739f5e5e17d4bae995ebfa1079c03b4f216c0ba57e9bd5"
Sep 06 14:23:09.642827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.642796    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:23:09.642877 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.642866    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:23:09.642914 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:09.642895    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 14:23:10.647716 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:10.647680    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerStarted","Data":"ade858e949cb02d06839e06567c3f5ecc6b1957586974c57217edfe81e37c688"}
Sep 06 14:23:11.139604 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:11.139551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:23:11.139976 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:11.139948    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 06 14:23:11.140099 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:11.140010    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 06 14:23:12.140388 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:12.140351    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 06 14:23:12.140822 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:12.140420    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 06 14:23:13.140405 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:13.140363    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 06 14:23:13.140827 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:13.140439    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 06 14:23:14.140779 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:14.140715    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 06 14:23:14.141173 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:14.140814    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 06 14:23:15.140011 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:15.139969    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 06 14:23:15.140287 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:15.140041    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 06 14:23:16.445482 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:16.445434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:23:16.446096 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:16.446077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:23:16.447168 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:16.447141    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:23:20.428991 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:20.428958    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s_588b0d32-5d88-4c87-9fbf-ba6ac65b5378/payload-processing/0.log"
Sep 06 14:23:32.884823 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.884783    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:23:32.888850 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.888825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:32.892102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.892076    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d46936\"/\"ipp-route-d46936-kserve-self-signed-certs\""
Sep 06 14:23:32.892252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.892112    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d46936\"/\"openshift-service-ca.crt\""
Sep 06 14:23:32.892252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.892076    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d46936\"/\"kube-root-ca.crt\""
Sep 06 14:23:32.892252 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.892135    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d46936\"/\"default-dockercfg-sqgjx\""
Sep 06 14:23:32.898734 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:32.898691    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:23:33.030363 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ckj\" (UniqueName: \"kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.030544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.030544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.030544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.030544 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.030693 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.030567    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131328 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ckj\" (UniqueName: \"kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131328 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131550 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131550 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131828 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131913 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.131967 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.131936    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.133833 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.133800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.133943 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.133926    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.138390 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.138343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ckj\" (UniqueName: \"kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj\") pod \"ipp-route-d46936-kserve-c7f69d75f-tn6hg\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") " pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.201336 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.201299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:33.472776 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.472729    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:23:33.475279 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:23:33.475244    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1da27a7b_17b4_4b93_8439_4d59cc338ddf.slice/crio-3c8b056f5c0fe42769a48a4671727018c694db5f48d1d1673c39dfecb6ae8110 WatchSource:0}: Error finding container 3c8b056f5c0fe42769a48a4671727018c694db5f48d1d1673c39dfecb6ae8110: Status 404 returned error can't find the container with id 3c8b056f5c0fe42769a48a4671727018c694db5f48d1d1673c39dfecb6ae8110
Sep 06 14:23:33.756567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.756476    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" event={"ID":"1da27a7b-17b4-4b93-8439-4d59cc338ddf","Type":"ContainerStarted","Data":"e5ef8470f4d8845eeae52a9205dacdf2f15f65d2d6636b919cb77037d36d0289"}
Sep 06 14:23:33.756567 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.756517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" event={"ID":"1da27a7b-17b4-4b93-8439-4d59cc338ddf","Type":"ContainerStarted","Data":"3c8b056f5c0fe42769a48a4671727018c694db5f48d1d1673c39dfecb6ae8110"}
Sep 06 14:23:33.771328 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:33.771272    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" podStartSLOduration=1.7712554900000002 podStartE2EDuration="1.77125549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:23:33.77017342 +0000 UTC m=+1550.796238596" watchObservedRunningTime="2026-09-06 14:23:33.77125549 +0000 UTC m=+1550.797320744"
Sep 06 14:23:43.201903 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.201863    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:43.202436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.201931    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:43.214618 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.214593    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:23:43.761236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.761199    2576 scope.go:117] "RemoveContainer" containerID="2bf47e4686fc584c1a38cd4dcc149f84c2c30e90f6431ace99675c8810753b76"
Sep 06 14:23:43.769472 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.769453    2576 scope.go:117] "RemoveContainer" containerID="082b9950ef74b0b1fc2b11f99267c50ef18e7f5b80469b33850be5114e150f9a"
Sep 06 14:23:43.815655 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:23:43.815627    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:24:01.013167 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:01.013129    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256_f03d0596-8350-4430-9c2e-c48e2a5daadf/payload-processing/0.log"
Sep 06 14:24:01.240081 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:01.240052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s_588b0d32-5d88-4c87-9fbf-ba6ac65b5378/payload-processing/0.log"
Sep 06 14:24:06.431644 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.431607    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:06.435047 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.435030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.437348 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.437324    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8e268e\""
Sep 06 14:24:06.450348 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.450314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:06.547022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.546979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.547231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.547123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkpj\" (UniqueName: \"kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.648514 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.648469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.648705 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.648554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkpj\" (UniqueName: \"kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.649254 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.649231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.656045 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.656024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkpj\" (UniqueName: \"kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj\") pod \"payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:06.744862 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:06.744741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:07.015180 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:07.015080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:07.018221 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:24:07.018191    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0bc500_4959_4526_841d_4d1e53bf19d4.slice/crio-823ee4a126be3dab369b2992b6e2c99a05ae401b1b8a039e144bd5ea7053d732 WatchSource:0}: Error finding container 823ee4a126be3dab369b2992b6e2c99a05ae401b1b8a039e144bd5ea7053d732: Status 404 returned error can't find the container with id 823ee4a126be3dab369b2992b6e2c99a05ae401b1b8a039e144bd5ea7053d732
Sep 06 14:24:07.916751 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:07.916711    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" event={"ID":"3c0bc500-4959-4526-841d-4d1e53bf19d4","Type":"ContainerStarted","Data":"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"}
Sep 06 14:24:07.917184 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:07.916790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" event={"ID":"3c0bc500-4959-4526-841d-4d1e53bf19d4","Type":"ContainerStarted","Data":"823ee4a126be3dab369b2992b6e2c99a05ae401b1b8a039e144bd5ea7053d732"}
Sep 06 14:24:07.917184 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:07.916904    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:07.931621 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:07.931570    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" podStartSLOduration=1.931552504 podStartE2EDuration="1.931552504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:07.930675003 +0000 UTC m=+1584.956740180" watchObservedRunningTime="2026-09-06 14:24:07.931552504 +0000 UTC m=+1584.957617730"
Sep 06 14:24:18.923891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:18.923859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:30.140331 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.140294    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"]
Sep 06 14:24:30.143702 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.143682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:30.145952 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.145925    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:24:30.145952 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.145940    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:24:30.150065 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.150038    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"]
Sep 06 14:24:30.170134 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.170099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv\" (UID: \"78c9c88a-74ce-4394-991f-92f600058aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:30.271202 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.271164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv\" (UID: \"78c9c88a-74ce-4394-991f-92f600058aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:30.271825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.271803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv\" (UID: \"78c9c88a-74ce-4394-991f-92f600058aa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:30.455576 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.455536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:30.723581 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:30.723557    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"]
Sep 06 14:24:30.725646 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:24:30.725612    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9c88a_74ce_4394_991f_92f600058aa1.slice/crio-3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17 WatchSource:0}: Error finding container 3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17: Status 404 returned error can't find the container with id 3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17
Sep 06 14:24:31.029946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:31.029846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv" event={"ID":"78c9c88a-74ce-4394-991f-92f600058aa1","Type":"ContainerStarted","Data":"fd09717b1b790094d5c16ed5dd32d6ec4e0b2f0549e3d4448477aae60db00c1c"}
Sep 06 14:24:31.029946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:31.029900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv" event={"ID":"78c9c88a-74ce-4394-991f-92f600058aa1","Type":"ContainerStarted","Data":"3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17"}
Sep 06 14:24:31.044515 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:31.044463    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv" podStartSLOduration=1.044448773 podStartE2EDuration="1.044448773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:31.043569942 +0000 UTC m=+1608.069635120" watchObservedRunningTime="2026-09-06 14:24:31.044448773 +0000 UTC m=+1608.070513948"
Sep 06 14:24:32.036263 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:32.036227    2576 generic.go:356] "Generic (PLEG): container finished" podID="78c9c88a-74ce-4394-991f-92f600058aa1" containerID="fd09717b1b790094d5c16ed5dd32d6ec4e0b2f0549e3d4448477aae60db00c1c" exitCode=0
Sep 06 14:24:32.036657 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:32.036301    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv" event={"ID":"78c9c88a-74ce-4394-991f-92f600058aa1","Type":"ContainerDied","Data":"fd09717b1b790094d5c16ed5dd32d6ec4e0b2f0549e3d4448477aae60db00c1c"}
Sep 06 14:24:33.171426 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:33.171395    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:33.191082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:33.191048    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca\") pod \"78c9c88a-74ce-4394-991f-92f600058aa1\" (UID: \"78c9c88a-74ce-4394-991f-92f600058aa1\") "
Sep 06 14:24:33.191456 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:33.191435    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78c9c88a-74ce-4394-991f-92f600058aa1" (UID: "78c9c88a-74ce-4394-991f-92f600058aa1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:24:33.291917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:33.291884    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9c88a-74ce-4394-991f-92f600058aa1-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:24:34.048286 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:34.048246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv" event={"ID":"78c9c88a-74ce-4394-991f-92f600058aa1","Type":"ContainerDied","Data":"3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17"}
Sep 06 14:24:34.048461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:34.048262    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"
Sep 06 14:24:34.048461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:34.048331    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eb0c57c04ba8bd6bc4d743a46b2e0ed6224a297ee5b6c6b914183f4f0950a17"
Sep 06 14:24:36.109025 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.108931    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:36.109455 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.109184    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" podUID="3c0bc500-4959-4526-841d-4d1e53bf19d4" containerName="payload-processing" containerID="cri-o://13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056" gracePeriod=30
Sep 06 14:24:36.249489 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.249423    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"]
Sep 06 14:24:36.252707 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.252675    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8e268e-dqnbv"]
Sep 06 14:24:36.380037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.380013    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:36.417679 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.417639    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume\") pod \"3c0bc500-4959-4526-841d-4d1e53bf19d4\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") "
Sep 06 14:24:36.417865 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.417699    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pkpj\" (UniqueName: \"kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj\") pod \"3c0bc500-4959-4526-841d-4d1e53bf19d4\" (UID: \"3c0bc500-4959-4526-841d-4d1e53bf19d4\") "
Sep 06 14:24:36.418127 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.418104    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c0bc500-4959-4526-841d-4d1e53bf19d4" (UID: "3c0bc500-4959-4526-841d-4d1e53bf19d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:24:36.419848 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.419825    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj" (OuterVolumeSpecName: "kube-api-access-7pkpj") pod "3c0bc500-4959-4526-841d-4d1e53bf19d4" (UID: "3c0bc500-4959-4526-841d-4d1e53bf19d4"). InnerVolumeSpecName "kube-api-access-7pkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:24:36.519376 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.519332    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c0bc500-4959-4526-841d-4d1e53bf19d4-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:24:36.519376 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:36.519368    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkpj\" (UniqueName: \"kubernetes.io/projected/3c0bc500-4959-4526-841d-4d1e53bf19d4-kube-api-access-7pkpj\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:24:37.067439 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.067399    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c0bc500-4959-4526-841d-4d1e53bf19d4" containerID="13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056" exitCode=0
Sep 06 14:24:37.067650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.067451    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"
Sep 06 14:24:37.067650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.067492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" event={"ID":"3c0bc500-4959-4526-841d-4d1e53bf19d4","Type":"ContainerDied","Data":"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"}
Sep 06 14:24:37.067650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.067555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n" event={"ID":"3c0bc500-4959-4526-841d-4d1e53bf19d4","Type":"ContainerDied","Data":"823ee4a126be3dab369b2992b6e2c99a05ae401b1b8a039e144bd5ea7053d732"}
Sep 06 14:24:37.067650 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.067588    2576 scope.go:117] "RemoveContainer" containerID="13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"
Sep 06 14:24:37.078533 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.078508    2576 scope.go:117] "RemoveContainer" containerID="13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"
Sep 06 14:24:37.078864 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:24:37.078840    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056\": container with ID starting with 13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056 not found: ID does not exist" containerID="13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"
Sep 06 14:24:37.078955 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.078873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056"} err="failed to get container status \"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056\": rpc error: code = NotFound desc = could not find container \"13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056\": container with ID starting with 13570ac901c529f5e6cee99496b1d79f8a29664f6e3b1e941b8f3f2a5cf24056 not found: ID does not exist"
Sep 06 14:24:37.089337 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.089311    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:37.090909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.090888    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-8e268e-6b544d7647-b6f6n"]
Sep 06 14:24:37.612195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.611235    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c0bc500-4959-4526-841d-4d1e53bf19d4" path="/var/lib/kubelet/pods/3c0bc500-4959-4526-841d-4d1e53bf19d4/volumes"
Sep 06 14:24:37.612195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:37.611815    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c9c88a-74ce-4394-991f-92f600058aa1" path="/var/lib/kubelet/pods/78c9c88a-74ce-4394-991f-92f600058aa1/volumes"
Sep 06 14:24:43.790946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:43.790908    2576 scope.go:117] "RemoveContainer" containerID="f3f51ccbe0144f5c945e96caa39779e27c9c1c1e144e1eff6dbdbd16668b9c60"
Sep 06 14:24:46.065889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.065852    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"]
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066177    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0bc500-4959-4526-841d-4d1e53bf19d4" containerName="payload-processing"
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066189    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0bc500-4959-4526-841d-4d1e53bf19d4" containerName="payload-processing"
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066199    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c9c88a-74ce-4394-991f-92f600058aa1" containerName="revoke-keys"
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066205    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c9c88a-74ce-4394-991f-92f600058aa1" containerName="revoke-keys"
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066266    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c9c88a-74ce-4394-991f-92f600058aa1" containerName="revoke-keys"
Sep 06 14:24:46.066303 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.066276    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0bc500-4959-4526-841d-4d1e53bf19d4" containerName="payload-processing"
Sep 06 14:24:46.069169 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.069150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:46.071386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.071361    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:24:46.071495 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.071398    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:24:46.075573 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.075546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"]
Sep 06 14:24:46.103889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.103848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d46936-xv578\" (UID: \"ff520704-0979-473e-99ac-62d9c10e59b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:46.205144 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.205103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d46936-xv578\" (UID: \"ff520704-0979-473e-99ac-62d9c10e59b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:46.205742 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.205719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d46936-xv578\" (UID: \"ff520704-0979-473e-99ac-62d9c10e59b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:46.381605 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.381510    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:46.652874 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:46.652825    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"]
Sep 06 14:24:46.655215 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:24:46.655186    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff520704_0979_473e_99ac_62d9c10e59b2.slice/crio-7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1 WatchSource:0}: Error finding container 7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1: Status 404 returned error can't find the container with id 7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1
Sep 06 14:24:47.121327 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:47.121291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578" event={"ID":"ff520704-0979-473e-99ac-62d9c10e59b2","Type":"ContainerStarted","Data":"3a571920dadfcd3f73530d40e2a739cba2ec66bba69c219b0b9e33b2ef704b3c"}
Sep 06 14:24:47.121686 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:47.121347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578" event={"ID":"ff520704-0979-473e-99ac-62d9c10e59b2","Type":"ContainerStarted","Data":"7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1"}
Sep 06 14:24:47.134284 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:47.134223    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578" podStartSLOduration=1.134205389 podStartE2EDuration="1.134205389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:24:47.133848885 +0000 UTC m=+1624.159914059" watchObservedRunningTime="2026-09-06 14:24:47.134205389 +0000 UTC m=+1624.160270565"
Sep 06 14:24:48.127570 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:48.127523    2576 generic.go:356] "Generic (PLEG): container finished" podID="ff520704-0979-473e-99ac-62d9c10e59b2" containerID="3a571920dadfcd3f73530d40e2a739cba2ec66bba69c219b0b9e33b2ef704b3c" exitCode=0
Sep 06 14:24:48.127969 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:48.127610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578" event={"ID":"ff520704-0979-473e-99ac-62d9c10e59b2","Type":"ContainerDied","Data":"3a571920dadfcd3f73530d40e2a739cba2ec66bba69c219b0b9e33b2ef704b3c"}
Sep 06 14:24:49.259710 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:49.259681    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:24:49.335151 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:49.335105    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca\") pod \"ff520704-0979-473e-99ac-62d9c10e59b2\" (UID: \"ff520704-0979-473e-99ac-62d9c10e59b2\") "
Sep 06 14:24:49.335522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:49.335500    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff520704-0979-473e-99ac-62d9c10e59b2" (UID: "ff520704-0979-473e-99ac-62d9c10e59b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:24:49.435821 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:49.435788    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff520704-0979-473e-99ac-62d9c10e59b2-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:24:50.137445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:50.137409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578" event={"ID":"ff520704-0979-473e-99ac-62d9c10e59b2","Type":"ContainerDied","Data":"7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1"}
Sep 06 14:24:50.137623 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:50.137455    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c746d016d6757e7be37200467d29e578fba770ef6c0eba71eb2e7d569a5aee1"
Sep 06 14:24:50.137623 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:24:50.137419    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"
Sep 06 14:25:01.621497 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.621453    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:25:01.622086 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.621792    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" podUID="f03d0596-8350-4430-9c2e-c48e2a5daadf" containerName="payload-processing" containerID="cri-o://57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312" gracePeriod=30
Sep 06 14:25:01.629213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.629171    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:25:01.629461 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.629428    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" podUID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" containerName="payload-pre-processing" containerID="cri-o://cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513" gracePeriod=30
Sep 06 14:25:01.764891 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.764822    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"]
Sep 06 14:25:01.768584 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.768551    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d46936-xv578"]
Sep 06 14:25:01.921122 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.921091    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:25:01.923200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:01.923180    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:25:02.060638 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.060592    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhrw7\" (UniqueName: \"kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7\") pod \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") "
Sep 06 14:25:02.060883 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.060649    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume\") pod \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\" (UID: \"553bb504-008c-4c47-aa06-3b2e8e46ba4a\") "
Sep 06 14:25:02.060883 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.060672    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume\") pod \"f03d0596-8350-4430-9c2e-c48e2a5daadf\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") "
Sep 06 14:25:02.060883 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.060728    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz9k8\" (UniqueName: \"kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8\") pod \"f03d0596-8350-4430-9c2e-c48e2a5daadf\" (UID: \"f03d0596-8350-4430-9c2e-c48e2a5daadf\") "
Sep 06 14:25:02.061112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.061073    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "553bb504-008c-4c47-aa06-3b2e8e46ba4a" (UID: "553bb504-008c-4c47-aa06-3b2e8e46ba4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:25:02.061233 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.061131    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f03d0596-8350-4430-9c2e-c48e2a5daadf" (UID: "f03d0596-8350-4430-9c2e-c48e2a5daadf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:25:02.062942 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.062921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8" (OuterVolumeSpecName: "kube-api-access-cz9k8") pod "f03d0596-8350-4430-9c2e-c48e2a5daadf" (UID: "f03d0596-8350-4430-9c2e-c48e2a5daadf"). InnerVolumeSpecName "kube-api-access-cz9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:25:02.063033 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.062950    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7" (OuterVolumeSpecName: "kube-api-access-fhrw7") pod "553bb504-008c-4c47-aa06-3b2e8e46ba4a" (UID: "553bb504-008c-4c47-aa06-3b2e8e46ba4a"). InnerVolumeSpecName "kube-api-access-fhrw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:25:02.161869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.161740    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9k8\" (UniqueName: \"kubernetes.io/projected/f03d0596-8350-4430-9c2e-c48e2a5daadf-kube-api-access-cz9k8\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:02.161869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.161814    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrw7\" (UniqueName: \"kubernetes.io/projected/553bb504-008c-4c47-aa06-3b2e8e46ba4a-kube-api-access-fhrw7\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:02.161869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.161823    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/553bb504-008c-4c47-aa06-3b2e8e46ba4a-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:02.161869 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.161832    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03d0596-8350-4430-9c2e-c48e2a5daadf-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:02.202573 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.202541    2576 generic.go:356] "Generic (PLEG): container finished" podID="f03d0596-8350-4430-9c2e-c48e2a5daadf" containerID="57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312" exitCode=0
Sep 06 14:25:02.202796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.202583    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"
Sep 06 14:25:02.202796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.202626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" event={"ID":"f03d0596-8350-4430-9c2e-c48e2a5daadf","Type":"ContainerDied","Data":"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"}
Sep 06 14:25:02.202796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.202675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256" event={"ID":"f03d0596-8350-4430-9c2e-c48e2a5daadf","Type":"ContainerDied","Data":"04c6bb5d395eb78ffefc5da0fee65c5643a01cc2b9b673f398fca5feb9709025"}
Sep 06 14:25:02.202796 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.202697    2576 scope.go:117] "RemoveContainer" containerID="57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"
Sep 06 14:25:02.204121 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.204100    2576 generic.go:356] "Generic (PLEG): container finished" podID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" containerID="cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513" exitCode=0
Sep 06 14:25:02.204234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.204123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" event={"ID":"553bb504-008c-4c47-aa06-3b2e8e46ba4a","Type":"ContainerDied","Data":"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"}
Sep 06 14:25:02.204234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.204137    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"
Sep 06 14:25:02.204234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.204176    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59" event={"ID":"553bb504-008c-4c47-aa06-3b2e8e46ba4a","Type":"ContainerDied","Data":"61d263b04c4f91233cd134eac8cba8f76e95018a2cf7f1a9ffb5058737b5d02d"}
Sep 06 14:25:02.212063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.212031    2576 scope.go:117] "RemoveContainer" containerID="57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"
Sep 06 14:25:02.212405 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:25:02.212382    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312\": container with ID starting with 57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312 not found: ID does not exist" containerID="57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"
Sep 06 14:25:02.212484 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.212414    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312"} err="failed to get container status \"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312\": rpc error: code = NotFound desc = could not find container \"57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312\": container with ID starting with 57a4cb16e094a49adb6b08d209e279dae6afdbaf7fbbedcc80ea4251c8e05312 not found: ID does not exist"
Sep 06 14:25:02.212484 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.212432    2576 scope.go:117] "RemoveContainer" containerID="cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"
Sep 06 14:25:02.221231 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.221211    2576 scope.go:117] "RemoveContainer" containerID="cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"
Sep 06 14:25:02.221556 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:25:02.221538    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513\": container with ID starting with cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513 not found: ID does not exist" containerID="cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"
Sep 06 14:25:02.221616 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.221567    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513"} err="failed to get container status \"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513\": rpc error: code = NotFound desc = could not find container \"cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513\": container with ID starting with cdf383f42d3092d04e1609b2a79a327fbeac9fdda3b13dcc5a33afefd6871513 not found: ID does not exist"
Sep 06 14:25:02.224176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.224148    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:25:02.227909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.227885    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d46936-558fbb7cbf-cm256"]
Sep 06 14:25:02.238110 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.238083    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:25:02.243672 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:02.243649    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d46936-66f964f786-zjt59"]
Sep 06 14:25:03.612004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:03.611972    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" path="/var/lib/kubelet/pods/553bb504-008c-4c47-aa06-3b2e8e46ba4a/volumes"
Sep 06 14:25:03.614512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:03.612934    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03d0596-8350-4430-9c2e-c48e2a5daadf" path="/var/lib/kubelet/pods/f03d0596-8350-4430-9c2e-c48e2a5daadf/volumes"
Sep 06 14:25:03.614512 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:03.613457    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff520704-0979-473e-99ac-62d9c10e59b2" path="/var/lib/kubelet/pods/ff520704-0979-473e-99ac-62d9c10e59b2/volumes"
Sep 06 14:25:07.102705 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:07.102666    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:25:22.122242 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.122185    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" podUID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" containerName="main" containerID="cri-o://e5ef8470f4d8845eeae52a9205dacdf2f15f65d2d6636b919cb77037d36d0289" gracePeriod=45
Sep 06 14:25:22.316339 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.316287    2576 generic.go:356] "Generic (PLEG): container finished" podID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" containerID="e5ef8470f4d8845eeae52a9205dacdf2f15f65d2d6636b919cb77037d36d0289" exitCode=0
Sep 06 14:25:22.316508 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.316356    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" event={"ID":"1da27a7b-17b4-4b93-8439-4d59cc338ddf","Type":"ContainerDied","Data":"e5ef8470f4d8845eeae52a9205dacdf2f15f65d2d6636b919cb77037d36d0289"}
Sep 06 14:25:22.360648 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.360620    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:25:22.449682 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449651    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.449682 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449687    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.449917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449733    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.449917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449808    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4ckj\" (UniqueName: \"kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.449917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449838    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.449917 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449883    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm\") pod \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\" (UID: \"1da27a7b-17b4-4b93-8439-4d59cc338ddf\") "
Sep 06 14:25:22.450137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449967    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:25:22.450137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.449989    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache" (OuterVolumeSpecName: "model-cache") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:25:22.450137 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.450058    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home" (OuterVolumeSpecName: "home") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:25:22.450299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.450208    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:22.450299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.450220    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:22.450299 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.450230    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:22.452410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.452380    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm" (OuterVolumeSpecName: "dshm") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:25:22.452526 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.452419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:25:22.452526 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.452494    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj" (OuterVolumeSpecName: "kube-api-access-n4ckj") pod "1da27a7b-17b4-4b93-8439-4d59cc338ddf" (UID: "1da27a7b-17b4-4b93-8439-4d59cc338ddf"). InnerVolumeSpecName "kube-api-access-n4ckj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:25:22.551316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.551270    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4ckj\" (UniqueName: \"kubernetes.io/projected/1da27a7b-17b4-4b93-8439-4d59cc338ddf-kube-api-access-n4ckj\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:22.551316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.551310    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1da27a7b-17b4-4b93-8439-4d59cc338ddf-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:22.551316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:22.551321    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1da27a7b-17b4-4b93-8439-4d59cc338ddf-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:23.321969 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.321943    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"
Sep 06 14:25:23.322390 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.321965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg" event={"ID":"1da27a7b-17b4-4b93-8439-4d59cc338ddf","Type":"ContainerDied","Data":"3c8b056f5c0fe42769a48a4671727018c694db5f48d1d1673c39dfecb6ae8110"}
Sep 06 14:25:23.322390 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.322021    2576 scope.go:117] "RemoveContainer" containerID="e5ef8470f4d8845eeae52a9205dacdf2f15f65d2d6636b919cb77037d36d0289"
Sep 06 14:25:23.341539 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.341510    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:25:23.344871 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.344844    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d46936/ipp-route-d46936-kserve-c7f69d75f-tn6hg"]
Sep 06 14:25:23.611006 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:23.610927    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" path="/var/lib/kubelet/pods/1da27a7b-17b4-4b93-8439-4d59cc338ddf/volumes"
Sep 06 14:25:50.532968 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.532933    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"]
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533243    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff520704-0979-473e-99ac-62d9c10e59b2" containerName="revoke-keys"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533256    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff520704-0979-473e-99ac-62d9c10e59b2" containerName="revoke-keys"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533272    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" containerName="main"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533277    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" containerName="main"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533283    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03d0596-8350-4430-9c2e-c48e2a5daadf" containerName="payload-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533290    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03d0596-8350-4430-9c2e-c48e2a5daadf" containerName="payload-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533296    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" containerName="payload-pre-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533302    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" containerName="payload-pre-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533348    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff520704-0979-473e-99ac-62d9c10e59b2" containerName="revoke-keys"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533357    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03d0596-8350-4430-9c2e-c48e2a5daadf" containerName="payload-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533363    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="553bb504-008c-4c47-aa06-3b2e8e46ba4a" containerName="payload-pre-processing"
Sep 06 14:25:50.533398 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.533369    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da27a7b-17b4-4b93-8439-4d59cc338ddf" containerName="main"
Sep 06 14:25:50.536112 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.536094    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:50.538371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.538348    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:25:50.538505 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.538348    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:25:50.542228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.542203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"]
Sep 06 14:25:50.713776 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.713718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx\" (UID: \"8db841b7-2011-46e8-b70a-126b47e2de9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:50.814692 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.814597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx\" (UID: \"8db841b7-2011-46e8-b70a-126b47e2de9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:50.815246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.815226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx\" (UID: \"8db841b7-2011-46e8-b70a-126b47e2de9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:50.847293 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:50.847262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:51.119404 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:51.119376    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"]
Sep 06 14:25:51.121476 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:25:51.121449    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db841b7_2011_46e8_b70a_126b47e2de9f.slice/crio-682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6 WatchSource:0}: Error finding container 682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6: Status 404 returned error can't find the container with id 682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6
Sep 06 14:25:51.459308 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:51.459275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx" event={"ID":"8db841b7-2011-46e8-b70a-126b47e2de9f","Type":"ContainerStarted","Data":"5dbaebdc61d4cec7cbeb53f6a06a5a585f7669cb22a0b2bb95b2065075e8b1f3"}
Sep 06 14:25:51.459477 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:51.459329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx" event={"ID":"8db841b7-2011-46e8-b70a-126b47e2de9f","Type":"ContainerStarted","Data":"682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6"}
Sep 06 14:25:51.474529 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:51.474462    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx" podStartSLOduration=1.474448484 podStartE2EDuration="1.474448484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:25:51.472553192 +0000 UTC m=+1688.498618368" watchObservedRunningTime="2026-09-06 14:25:51.474448484 +0000 UTC m=+1688.500513659"
Sep 06 14:25:52.466036 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:52.466000    2576 generic.go:356] "Generic (PLEG): container finished" podID="8db841b7-2011-46e8-b70a-126b47e2de9f" containerID="5dbaebdc61d4cec7cbeb53f6a06a5a585f7669cb22a0b2bb95b2065075e8b1f3" exitCode=0
Sep 06 14:25:52.466410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:52.466055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx" event={"ID":"8db841b7-2011-46e8-b70a-126b47e2de9f","Type":"ContainerDied","Data":"5dbaebdc61d4cec7cbeb53f6a06a5a585f7669cb22a0b2bb95b2065075e8b1f3"}
Sep 06 14:25:53.601527 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:53.601499    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:53.742752 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:53.742652    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca\") pod \"8db841b7-2011-46e8-b70a-126b47e2de9f\" (UID: \"8db841b7-2011-46e8-b70a-126b47e2de9f\") "
Sep 06 14:25:53.743072 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:53.743050    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8db841b7-2011-46e8-b70a-126b47e2de9f" (UID: "8db841b7-2011-46e8-b70a-126b47e2de9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:25:53.844306 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:53.844267    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db841b7-2011-46e8-b70a-126b47e2de9f-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:25:54.478256 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:54.478226    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"
Sep 06 14:25:54.478464 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:54.478233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx" event={"ID":"8db841b7-2011-46e8-b70a-126b47e2de9f","Type":"ContainerDied","Data":"682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6"}
Sep 06 14:25:54.478464 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:25:54.478347    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682a4f6bdc046fb8ccb39f559fc1de7bcdb0d72992b15acce558019dd11be3a6"
Sep 06 14:26:00.992667 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:00.992616    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:26:00.993107 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:00.992929    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" podUID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" containerName="payload-processing" containerID="cri-o://1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d" gracePeriod=30
Sep 06 14:26:01.131244 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.131208    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"]
Sep 06 14:26:01.133640 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.133603    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c1a83e-c4hsx"]
Sep 06 14:26:01.265541 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.265513    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:26:01.311423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.311392    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume\") pod \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") "
Sep 06 14:26:01.311617 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.311491    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmnwn\" (UniqueName: \"kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn\") pod \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\" (UID: \"588b0d32-5d88-4c87-9fbf-ba6ac65b5378\") "
Sep 06 14:26:01.311773 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.311732    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "588b0d32-5d88-4c87-9fbf-ba6ac65b5378" (UID: "588b0d32-5d88-4c87-9fbf-ba6ac65b5378"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:01.313670 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.313645    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn" (OuterVolumeSpecName: "kube-api-access-nmnwn") pod "588b0d32-5d88-4c87-9fbf-ba6ac65b5378" (UID: "588b0d32-5d88-4c87-9fbf-ba6ac65b5378"). InnerVolumeSpecName "kube-api-access-nmnwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:01.413173 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.413134    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-plugins-config-volume\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:01.413173 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.413166    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmnwn\" (UniqueName: \"kubernetes.io/projected/588b0d32-5d88-4c87-9fbf-ba6ac65b5378-kube-api-access-nmnwn\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:01.512694 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.512606    2576 generic.go:356] "Generic (PLEG): container finished" podID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" containerID="1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d" exitCode=0
Sep 06 14:26:01.512694 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.512651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"
Sep 06 14:26:01.512694 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.512672    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" event={"ID":"588b0d32-5d88-4c87-9fbf-ba6ac65b5378","Type":"ContainerDied","Data":"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"}
Sep 06 14:26:01.512988 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.512714    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s" event={"ID":"588b0d32-5d88-4c87-9fbf-ba6ac65b5378","Type":"ContainerDied","Data":"1a7df2babc8340b6f71997e57c0cd6ea3b36a39c5e53bdb107cd336094a7903c"}
Sep 06 14:26:01.512988 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.512738    2576 scope.go:117] "RemoveContainer" containerID="1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"
Sep 06 14:26:01.521594 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.521576    2576 scope.go:117] "RemoveContainer" containerID="1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"
Sep 06 14:26:01.521917 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:26:01.521895    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d\": container with ID starting with 1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d not found: ID does not exist" containerID="1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"
Sep 06 14:26:01.521975 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.521927    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d"} err="failed to get container status \"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d\": rpc error: code = NotFound desc = could not find container \"1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d\": container with ID starting with 1643759f9e6fb5757bc5583f909f1fc588a2bf646670f12b1a2d151567ce790d not found: ID does not exist"
Sep 06 14:26:01.534022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.533994    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:26:01.538040 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.538018    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c1a83e-779f4fc6b9-v746s"]
Sep 06 14:26:01.610415 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.610385    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" path="/var/lib/kubelet/pods/588b0d32-5d88-4c87-9fbf-ba6ac65b5378/volumes"
Sep 06 14:26:01.610756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:01.610743    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db841b7-2011-46e8-b70a-126b47e2de9f" path="/var/lib/kubelet/pods/8db841b7-2011-46e8-b70a-126b47e2de9f/volumes"
Sep 06 14:26:07.417119 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.417038    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:26:07.417552 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.417292    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy" containerID="cri-o://ade858e949cb02d06839e06567c3f5ecc6b1957586974c57217edfe81e37c688" gracePeriod=30
Sep 06 14:26:07.769080 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.768988    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"]
Sep 06 14:26:07.769310 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" containerName="payload-processing"
Sep 06 14:26:07.769359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769311    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" containerName="payload-processing"
Sep 06 14:26:07.769359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769322    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db841b7-2011-46e8-b70a-126b47e2de9f" containerName="revoke-keys"
Sep 06 14:26:07.769359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769328    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db841b7-2011-46e8-b70a-126b47e2de9f" containerName="revoke-keys"
Sep 06 14:26:07.769453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769380    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="588b0d32-5d88-4c87-9fbf-ba6ac65b5378" containerName="payload-processing"
Sep 06 14:26:07.769453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.769391    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db841b7-2011-46e8-b70a-126b47e2de9f" containerName="revoke-keys"
Sep 06 14:26:07.773513 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.773492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:07.775742 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.775717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:26:07.775881 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.775717    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:26:07.778861 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.778834    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"]
Sep 06 14:26:07.871591 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.871546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5\" (UID: \"815abf7c-7e40-4c3c-8e96-81c3adb25324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:07.972685 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.972647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5\" (UID: \"815abf7c-7e40-4c3c-8e96-81c3adb25324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:07.973279 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:07.973257    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5\" (UID: \"815abf7c-7e40-4c3c-8e96-81c3adb25324\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:08.085013 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:08.084920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:08.352806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:08.352686    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"]
Sep 06 14:26:08.355661 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:26:08.355620    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815abf7c_7e40_4c3c_8e96_81c3adb25324.slice/crio-3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f WatchSource:0}: Error finding container 3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f: Status 404 returned error can't find the container with id 3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f
Sep 06 14:26:08.550671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:08.550634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5" event={"ID":"815abf7c-7e40-4c3c-8e96-81c3adb25324","Type":"ContainerStarted","Data":"3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f"}
Sep 06 14:26:09.556190 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:09.556155    2576 generic.go:356] "Generic (PLEG): container finished" podID="815abf7c-7e40-4c3c-8e96-81c3adb25324" containerID="4fb261a1f32bb73b418b729832eb5e46532f1e17d83d4c670c8421a8177e261c" exitCode=0
Sep 06 14:26:09.556559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:09.556219    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5" event={"ID":"815abf7c-7e40-4c3c-8e96-81c3adb25324","Type":"ContainerDied","Data":"4fb261a1f32bb73b418b729832eb5e46532f1e17d83d4c670c8421a8177e261c"}
Sep 06 14:26:10.692052 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:10.692020    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:10.800281 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:10.800246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca\") pod \"815abf7c-7e40-4c3c-8e96-81c3adb25324\" (UID: \"815abf7c-7e40-4c3c-8e96-81c3adb25324\") "
Sep 06 14:26:10.800666 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:10.800646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "815abf7c-7e40-4c3c-8e96-81c3adb25324" (UID: "815abf7c-7e40-4c3c-8e96-81c3adb25324"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:10.901382 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:10.901292    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/815abf7c-7e40-4c3c-8e96-81c3adb25324-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:11.567186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:11.567153    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"
Sep 06 14:26:11.567186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:11.567177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5" event={"ID":"815abf7c-7e40-4c3c-8e96-81c3adb25324","Type":"ContainerDied","Data":"3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f"}
Sep 06 14:26:11.567469 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:11.567249    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6aa5ba6212d55edabc0b83593b59ce8ac1d5729f7b73027616b18a423e1c8f"
Sep 06 14:26:12.574205 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.574170    2576 generic.go:356] "Generic (PLEG): container finished" podID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerID="ade858e949cb02d06839e06567c3f5ecc6b1957586974c57217edfe81e37c688" exitCode=0
Sep 06 14:26:12.574607 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.574244    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerDied","Data":"ade858e949cb02d06839e06567c3f5ecc6b1957586974c57217edfe81e37c688"}
Sep 06 14:26:12.574607 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.574314    2576 scope.go:117] "RemoveContainer" containerID="023976c3d8c2a60c02739f5e5e17d4bae995ebfa1079c03b4f216c0ba57e9bd5"
Sep 06 14:26:12.670140 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.670109    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:26:12.717678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717573    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.717678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717611    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.717678 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717653    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717685    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717793    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717842    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717891    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717918    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9qvr\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr\") pod \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\" (UID: \"f22292ba-6cdb-4d4f-8074-ca6384537ab0\") "
Sep 06 14:26:12.718004 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.717914    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718054    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718066    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718177    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718271    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718291    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718305    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.718362 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718320    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.718712 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.718385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:12.720162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.720135    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:12.720302 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.720186    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:12.720302 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.720182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr" (OuterVolumeSpecName: "kube-api-access-j9qvr") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "kube-api-access-j9qvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:12.720302 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.720277    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f22292ba-6cdb-4d4f-8074-ca6384537ab0" (UID: "f22292ba-6cdb-4d4f-8074-ca6384537ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:26:12.819453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.819415    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.819453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.819449    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qvr\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-kube-api-access-j9qvr\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.819453 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.819463    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.819704 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.819476    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:12.819704 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:12.819486    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22292ba-6cdb-4d4f-8074-ca6384537ab0-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:13.580142 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.580104    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"
Sep 06 14:26:13.580525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.580151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr" event={"ID":"f22292ba-6cdb-4d4f-8074-ca6384537ab0","Type":"ContainerDied","Data":"185f12e1b0608179c39b0474b18c926b555bc89d43e05c842131f54cd598482f"}
Sep 06 14:26:13.580525 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.580213    2576 scope.go:117] "RemoveContainer" containerID="ade858e949cb02d06839e06567c3f5ecc6b1957586974c57217edfe81e37c688"
Sep 06 14:26:13.599882 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.599850    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:26:13.603909 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.603878    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c1a83e-openshift-default-5d74564dc4-6j7fr"]
Sep 06 14:26:13.609456 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:13.609431    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" path="/var/lib/kubelet/pods/f22292ba-6cdb-4d4f-8074-ca6384537ab0/volumes"
Sep 06 14:26:18.433532 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:18.433491    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"]
Sep 06 14:26:18.437821 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:18.437795    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0aaff3-qkkx5"]
Sep 06 14:26:19.609728 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:19.609692    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="815abf7c-7e40-4c3c-8e96-81c3adb25324" path="/var/lib/kubelet/pods/815abf7c-7e40-4c3c-8e96-81c3adb25324/volumes"
Sep 06 14:26:24.785706 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:24.785672    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:26:24.786148 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:24.785918    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy" containerID="cri-o://aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c" gracePeriod=30
Sep 06 14:26:25.138096 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138012    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"]
Sep 06 14:26:25.138322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138310    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:26:25.138370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138323    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:26:25.138370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138331    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:26:25.138370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138336    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:26:25.138370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138352    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815abf7c-7e40-4c3c-8e96-81c3adb25324" containerName="revoke-keys"
Sep 06 14:26:25.138370 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138357    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="815abf7c-7e40-4c3c-8e96-81c3adb25324" containerName="revoke-keys"
Sep 06 14:26:25.138531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138399    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:26:25.138531 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.138408    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="815abf7c-7e40-4c3c-8e96-81c3adb25324" containerName="revoke-keys"
Sep 06 14:26:25.141361 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.141343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:25.143414 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.143387    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:26:25.143564 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.143545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 14:26:25.147430 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.147401    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"]
Sep 06 14:26:25.233711 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.233675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl\" (UID: \"5ae25d12-3a1f-400f-8a88-0e62d5669f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:25.335194 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.335151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl\" (UID: \"5ae25d12-3a1f-400f-8a88-0e62d5669f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:25.335821 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.335799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl\" (UID: \"5ae25d12-3a1f-400f-8a88-0e62d5669f97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:25.453613 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.453575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:25.721213 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:25.721188    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"]
Sep 06 14:26:25.724050 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:26:25.724023    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae25d12_3a1f_400f_8a88_0e62d5669f97.slice/crio-b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd WatchSource:0}: Error finding container b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd: Status 404 returned error can't find the container with id b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd
Sep 06 14:26:26.647778 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:26.647726    2576 generic.go:356] "Generic (PLEG): container finished" podID="5ae25d12-3a1f-400f-8a88-0e62d5669f97" containerID="50dc3a522b246a78db641fec3697a24a4bf4fcdb173291f8f1b2a1afc67d5afd" exitCode=0
Sep 06 14:26:26.648142 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:26.647786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl" event={"ID":"5ae25d12-3a1f-400f-8a88-0e62d5669f97","Type":"ContainerDied","Data":"50dc3a522b246a78db641fec3697a24a4bf4fcdb173291f8f1b2a1afc67d5afd"}
Sep 06 14:26:26.648142 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:26.647844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl" event={"ID":"5ae25d12-3a1f-400f-8a88-0e62d5669f97","Type":"ContainerStarted","Data":"b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd"}
Sep 06 14:26:27.774635 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:27.774608    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:27.958683 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:27.958648    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca\") pod \"5ae25d12-3a1f-400f-8a88-0e62d5669f97\" (UID: \"5ae25d12-3a1f-400f-8a88-0e62d5669f97\") "
Sep 06 14:26:27.959044 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:27.959021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ae25d12-3a1f-400f-8a88-0e62d5669f97" (UID: "5ae25d12-3a1f-400f-8a88-0e62d5669f97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:28.060029 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:28.059989    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae25d12-3a1f-400f-8a88-0e62d5669f97-maas-api-service-ca\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:28.657986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:28.657953    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"
Sep 06 14:26:28.657986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:28.657977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl" event={"ID":"5ae25d12-3a1f-400f-8a88-0e62d5669f97","Type":"ContainerDied","Data":"b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd"}
Sep 06 14:26:28.658250 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:28.658023    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b121c582fde25bdd2b611150c8211f5eaf4dea6f6cb19a3300bd66bbd16192dd"
Sep 06 14:26:30.024253 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.024226    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:26:30.179354 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cng5\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179415    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179486    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179522 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179515    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179841 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179540    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179841 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179581    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179841 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179614    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket\") pod \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\" (UID: \"e49f66d2-13bb-4201-a8d9-d7e43a7a8543\") "
Sep 06 14:26:30.179841 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179798    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data" (OuterVolumeSpecName: "istio-data") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:30.180063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179976    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.180063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.179988    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:30.180063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.180003    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:30.180063 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.180004    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:30.180268 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.180159    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:30.181792 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.181745    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:26:30.181792 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.181779    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token" (OuterVolumeSpecName: "istio-token") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:30.182061 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.182034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:30.182150 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.182038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5" (OuterVolumeSpecName: "kube-api-access-2cng5") pod "e49f66d2-13bb-4201-a8d9-d7e43a7a8543" (UID: "e49f66d2-13bb-4201-a8d9-d7e43a7a8543"). InnerVolumeSpecName "kube-api-access-2cng5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:30.281316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281280    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281311    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281322    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cng5\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-kube-api-access-2cng5\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281332    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281341    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281348    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281355    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.281557 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.281363    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49f66d2-13bb-4201-a8d9-d7e43a7a8543-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:30.668836 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.668804    2576 generic.go:356] "Generic (PLEG): container finished" podID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerID="aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c" exitCode=0
Sep 06 14:26:30.669010 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.668855    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"
Sep 06 14:26:30.669010 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.668899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerDied","Data":"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"}
Sep 06 14:26:30.669010 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.668954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl" event={"ID":"e49f66d2-13bb-4201-a8d9-d7e43a7a8543","Type":"ContainerDied","Data":"cd80643896c8ef6e15718e7c3d471583e98fde2c1d0a015f88fedeaaccf74a97"}
Sep 06 14:26:30.669010 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.668977    2576 scope.go:117] "RemoveContainer" containerID="aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"
Sep 06 14:26:30.678294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.678277    2576 scope.go:117] "RemoveContainer" containerID="71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"
Sep 06 14:26:30.686079 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.686060    2576 scope.go:117] "RemoveContainer" containerID="aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"
Sep 06 14:26:30.686359 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:26:30.686340    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c\": container with ID starting with aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c not found: ID does not exist" containerID="aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"
Sep 06 14:26:30.686423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.686366    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c"} err="failed to get container status \"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c\": rpc error: code = NotFound desc = could not find container \"aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c\": container with ID starting with aaa5a37a0ee4c18780e48acfc1d1ffd4ac05e5c781eccfd42be7db3b525d5a7c not found: ID does not exist"
Sep 06 14:26:30.686423 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.686382    2576 scope.go:117] "RemoveContainer" containerID="71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"
Sep 06 14:26:30.686674 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:26:30.686653    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda\": container with ID starting with 71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda not found: ID does not exist" containerID="71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"
Sep 06 14:26:30.686739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.686680    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda"} err="failed to get container status \"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda\": rpc error: code = NotFound desc = could not find container \"71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda\": container with ID starting with 71de8fc5e652c1328fab74030dad58f7139f6e6ff827a7795fcc512f1784edda not found: ID does not exist"
Sep 06 14:26:30.689369 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.689345    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:26:30.692491 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:30.692466    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0aaff3-openshift-default-5b9f4dc4c8-x9cdl"]
Sep 06 14:26:31.609875 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:31.609842    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" path="/var/lib/kubelet/pods/e49f66d2-13bb-4201-a8d9-d7e43a7a8543/volumes"
Sep 06 14:26:35.748901 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:35.748864    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"]
Sep 06 14:26:35.752859 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:35.752824    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c443d8-w8qdl"]
Sep 06 14:26:37.609051 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:37.609017    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae25d12-3a1f-400f-8a88-0e62d5669f97" path="/var/lib/kubelet/pods/5ae25d12-3a1f-400f-8a88-0e62d5669f97/volumes"
Sep 06 14:26:42.007677 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:42.007642    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:26:42.008240 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:42.007894    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy" containerID="cri-o://369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17" gracePeriod=30
Sep 06 14:26:43.834578 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:43.834545    2576 scope.go:117] "RemoveContainer" containerID="a367536f7b890750fde12aa707818962ebf711a86d58ec7f7c4c8565fad1cbb2"
Sep 06 14:26:47.262130 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.262102    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:26:47.311446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311360    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311414    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311446 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311442    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311476    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311501    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311527    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311550    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311587    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7rv6\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.311724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311619    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy\") pod \"28489758-86e0-493b-bb12-8e0ee99366da\" (UID: \"28489758-86e0-493b-bb12-8e0ee99366da\") "
Sep 06 14:26:47.312082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311857    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:47.312082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311915    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:47.312082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.311936    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:47.312082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.312006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data" (OuterVolumeSpecName: "istio-data") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:47.312082 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.312009    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 14:26:47.313608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.313585    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 06 14:26:47.313981 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.313954    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:26:47.314261 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.314236    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token" (OuterVolumeSpecName: "istio-token") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:47.314327 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.314294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6" (OuterVolumeSpecName: "kube-api-access-g7rv6") pod "28489758-86e0-493b-bb12-8e0ee99366da" (UID: "28489758-86e0-493b-bb12-8e0ee99366da"). InnerVolumeSpecName "kube-api-access-g7rv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:26:47.412486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412446    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7rv6\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-kube-api-access-g7rv6\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412480    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-envoy\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412493    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-istio-data\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412504    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28489758-86e0-493b-bb12-8e0ee99366da-istio-podinfo\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412515    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412526    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28489758-86e0-493b-bb12-8e0ee99366da-istiod-ca-cert\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412536    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28489758-86e0-493b-bb12-8e0ee99366da-istio-token\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412546    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-workload-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.412743 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.412556    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28489758-86e0-493b-bb12-8e0ee99366da-credential-socket\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:26:47.756150 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.756115    2576 generic.go:356] "Generic (PLEG): container finished" podID="28489758-86e0-493b-bb12-8e0ee99366da" containerID="369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17" exitCode=0
Sep 06 14:26:47.756316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.756162    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"
Sep 06 14:26:47.756316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.756198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerDied","Data":"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"}
Sep 06 14:26:47.756316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.756253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb" event={"ID":"28489758-86e0-493b-bb12-8e0ee99366da","Type":"ContainerDied","Data":"8b6907772c377d13e97b8d3d07ccf103090149be2f08ee381bfb792ec162b4af"}
Sep 06 14:26:47.756316 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.756282    2576 scope.go:117] "RemoveContainer" containerID="369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"
Sep 06 14:26:47.765006 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.764986    2576 scope.go:117] "RemoveContainer" containerID="16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"
Sep 06 14:26:47.772534 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.772506    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:26:47.773391 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.773369    2576 scope.go:117] "RemoveContainer" containerID="369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"
Sep 06 14:26:47.773729 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:26:47.773706    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17\": container with ID starting with 369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17 not found: ID does not exist" containerID="369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"
Sep 06 14:26:47.773791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.773742    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17"} err="failed to get container status \"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17\": rpc error: code = NotFound desc = could not find container \"369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17\": container with ID starting with 369131ab042d8f9282409f8c000a213818300f5e5b1c31e097bd700355008b17 not found: ID does not exist"
Sep 06 14:26:47.773791 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.773776    2576 scope.go:117] "RemoveContainer" containerID="16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"
Sep 06 14:26:47.774060 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:26:47.774042    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a\": container with ID starting with 16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a not found: ID does not exist" containerID="16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"
Sep 06 14:26:47.774111 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.774066    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a"} err="failed to get container status \"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a\": rpc error: code = NotFound desc = could not find container \"16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a\": container with ID starting with 16b4661ca9edc9bf8bdff5537689c8a8fa68345e2f5b8a2f1141946931b9422a not found: ID does not exist"
Sep 06 14:26:47.777454 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:47.777428    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c443d8-openshift-default-84cb85fc4b-8n2xb"]
Sep 06 14:26:49.609749 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:26:49.609705    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28489758-86e0-493b-bb12-8e0ee99366da" path="/var/lib/kubelet/pods/28489758-86e0-493b-bb12-8e0ee99366da/volumes"
Sep 06 14:27:43.537756 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:27:43.537729    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:27:43.682132 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:27:43.682106    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:27:43.863176 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:27:43.863082    2576 scope.go:117] "RemoveContainer" containerID="bdb45a00783fc2225d815fabeb5a8de19ff634edbff420b09c55a34ea3e087a3"
Sep 06 14:27:43.870966 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:27:43.870948    2576 scope.go:117] "RemoveContainer" containerID="17356aa1512a3f82ad3b84ff0077afecb81e060ced72a4ba87065c3862a5a822"
Sep 06 14:27:43.878234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:27:43.878213    2576 scope.go:117] "RemoveContainer" containerID="c7061c5ff5a7773f064f4841a7e8bb6a4f6c63409496ed50bf420fc8ee99bb72"
Sep 06 14:30:00.136079 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136039    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"]
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136337    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136351    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136362    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136368    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136382    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136388    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136401    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae25d12-3a1f-400f-8a88-0e62d5669f97" containerName="revoke-keys"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136406    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae25d12-3a1f-400f-8a88-0e62d5669f97" containerName="revoke-keys"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136464    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae25d12-3a1f-400f-8a88-0e62d5669f97" containerName="revoke-keys"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136474    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22292ba-6cdb-4d4f-8074-ca6384537ab0" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136483    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136489    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136495    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="28489758-86e0-493b-bb12-8e0ee99366da" containerName="istio-proxy"
Sep 06 14:30:00.136571 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.136502    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:30:00.139445 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.139428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"
Sep 06 14:30:00.141577 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.141549    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-c6dpp\""
Sep 06 14:30:00.145218 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.145185    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"]
Sep 06 14:30:00.149925 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.149902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"
Sep 06 14:30:00.417948 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.417882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"]
Sep 06 14:30:00.420612 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:30:00.420580    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4eb07e4_0987_4b47_a632_7c5c4317ffc7.slice/crio-a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203 WatchSource:0}: Error finding container a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203: Status 404 returned error can't find the container with id a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203
Sep 06 14:30:00.597913 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:00.597875    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc" event={"ID":"c4eb07e4-0987-4b47-a632-7c5c4317ffc7","Type":"ContainerStarted","Data":"a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203"}
Sep 06 14:30:01.603812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:01.603703    2576 generic.go:356] "Generic (PLEG): container finished" podID="c4eb07e4-0987-4b47-a632-7c5c4317ffc7" containerID="ed93449b118bd7aa8a26acd30482df4cedcf9cdba4ba62350cecd5dcb022abaf" exitCode=0
Sep 06 14:30:01.603812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:01.603795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc" event={"ID":"c4eb07e4-0987-4b47-a632-7c5c4317ffc7","Type":"ContainerDied","Data":"ed93449b118bd7aa8a26acd30482df4cedcf9cdba4ba62350cecd5dcb022abaf"}
Sep 06 14:30:02.732671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:02.732639    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"
Sep 06 14:30:03.612809 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:03.612778    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc"
Sep 06 14:30:03.612977 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:03.612781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29811750-cmrbc" event={"ID":"c4eb07e4-0987-4b47-a632-7c5c4317ffc7","Type":"ContainerDied","Data":"a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203"}
Sep 06 14:30:03.612977 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:03.612905    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f91c176d73fa446612afeefddc2247446d8d934ea154d3ab078b931a731203"
Sep 06 14:30:43.917403 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:30:43.917372    2576 scope.go:117] "RemoveContainer" containerID="fd09717b1b790094d5c16ed5dd32d6ec4e0b2f0549e3d4448477aae60db00c1c"
Sep 06 14:31:11.590608 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.590571    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:31:11.591085 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.591066    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:31:11.591129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.591087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e49f66d2-13bb-4201-a8d9-d7e43a7a8543" containerName="istio-proxy"
Sep 06 14:31:11.591129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.591107    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4eb07e4-0987-4b47-a632-7c5c4317ffc7" containerName="cleanup"
Sep 06 14:31:11.591129 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.591115    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4eb07e4-0987-4b47-a632-7c5c4317ffc7" containerName="cleanup"
Sep 06 14:31:11.591240 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.591216    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4eb07e4-0987-4b47-a632-7c5c4317ffc7" containerName="cleanup"
Sep 06 14:31:11.594449 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.594431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.596536 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.596512    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fda7e6e6-kserve-self-signed-certs\""
Sep 06 14:31:11.602607 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.602579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:31:11.767771 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.767946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.767946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767899    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.767946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2jq\" (UniqueName: \"kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.768070 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.768070 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.767989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869195 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2jq\" (UniqueName: \"kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869490 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869490 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869490 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869270    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.869663 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.869659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.871468 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.871443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.871656 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.871637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.876022 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.875996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2jq\" (UniqueName: \"kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq\") pod \"e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") " pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:11.904246 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:11.904212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:12.176565 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:12.176529    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:31:12.179625 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:31:12.179598    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5505e4ac_d7e3_4bf1_9d32_286529be0123.slice/crio-4b5b30c19872212d66d2cef12686614d051e241f6c507f402e75a19c68d40285 WatchSource:0}: Error finding container 4b5b30c19872212d66d2cef12686614d051e241f6c507f402e75a19c68d40285: Status 404 returned error can't find the container with id 4b5b30c19872212d66d2cef12686614d051e241f6c507f402e75a19c68d40285
Sep 06 14:31:12.906774 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:12.906738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" event={"ID":"5505e4ac-d7e3-4bf1-9d32-286529be0123","Type":"ContainerStarted","Data":"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"}
Sep 06 14:31:12.907186 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:12.906798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" event={"ID":"5505e4ac-d7e3-4bf1-9d32-286529be0123","Type":"ContainerStarted","Data":"4b5b30c19872212d66d2cef12686614d051e241f6c507f402e75a19c68d40285"}
Sep 06 14:31:12.920387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:12.920334    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" podStartSLOduration=1.9203155330000001 podStartE2EDuration="1.920315533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:31:12.920044716 +0000 UTC m=+2009.946109917" watchObservedRunningTime="2026-09-06 14:31:12.920315533 +0000 UTC m=+2009.946380710"
Sep 06 14:31:21.904632 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:21.904595    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:21.905209 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:21.904655    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:21.917413 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:21.917385    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:21.956242 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:21.956215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:31:43.936307 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:43.936267    2576 scope.go:117] "RemoveContainer" containerID="3a571920dadfcd3f73530d40e2a739cba2ec66bba69c219b0b9e33b2ef704b3c"
Sep 06 14:31:45.082322 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.082282    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:31:45.086568 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.086549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.088745 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.088722    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fda7e6e6-kserve-self-signed-certs\""
Sep 06 14:31:45.092601 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.092579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:31:45.149947 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.149904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.150159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.149989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.150159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.150079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.150159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.150116    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.150159 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.150147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.150380 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.150186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6r9\" (UniqueName: \"kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251491    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251538 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251849 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251849 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251849 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.251849 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251636    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6r9\" (UniqueName: \"kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.252048 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.252048 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.251992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.252048 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.252033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.253946 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.253920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.254202 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.254183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.264236 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.264210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6r9\" (UniqueName: \"kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9\") pod \"e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") " pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.396951 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.396861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:45.665960 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:45.665926    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:31:45.668997 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:31:45.668970    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71212f4_5905_4cb6_8fd9_b43fe1166804.slice/crio-9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863 WatchSource:0}: Error finding container 9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863: Status 404 returned error can't find the container with id 9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863
Sep 06 14:31:46.048410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:46.048318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" event={"ID":"b71212f4-5905-4cb6-8fd9-b43fe1166804","Type":"ContainerStarted","Data":"396182e5b8c9d7667ff720af6bfcf019b6b8d018329316a7d045c24eaa1f5a63"}
Sep 06 14:31:46.048410 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:46.048367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" event={"ID":"b71212f4-5905-4cb6-8fd9-b43fe1166804","Type":"ContainerStarted","Data":"9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863"}
Sep 06 14:31:46.064530 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:46.064466    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" podStartSLOduration=1.064445275 podStartE2EDuration="1.064445275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:31:46.062626064 +0000 UTC m=+2043.088691242" watchObservedRunningTime="2026-09-06 14:31:46.064445275 +0000 UTC m=+2043.090510450"
Sep 06 14:31:55.397480 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:55.397433    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:55.398011 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:55.397493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:55.410234 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:55.410202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:31:56.102237 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:31:56.102206    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:32:20.376297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:20.376207    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:32:22.005084 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:22.005046    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:32:35.396037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.395996    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" podUID="5505e4ac-d7e3-4bf1-9d32-286529be0123" containerName="main" containerID="cri-o://b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3" gracePeriod=45
Sep 06 14:32:35.639595 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.639565    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:32:35.800359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800321    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj2jq\" (UniqueName: \"kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800401    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800439    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800465    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800637 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800487    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir\") pod \"5505e4ac-d7e3-4bf1-9d32-286529be0123\" (UID: \"5505e4ac-d7e3-4bf1-9d32-286529be0123\") "
Sep 06 14:32:35.800889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache" (OuterVolumeSpecName: "model-cache") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:35.800889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800724    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home" (OuterVolumeSpecName: "home") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:35.800889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800824    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:35.800889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800842    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:35.800889 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.800862    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:35.802568 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.802538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm" (OuterVolumeSpecName: "dshm") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:35.802670 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.802608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:32:35.802670 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.802647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq" (OuterVolumeSpecName: "kube-api-access-gj2jq") pod "5505e4ac-d7e3-4bf1-9d32-286529be0123" (UID: "5505e4ac-d7e3-4bf1-9d32-286529be0123"). InnerVolumeSpecName "kube-api-access-gj2jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:32:35.901825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.901783    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:35.901825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.901819    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5505e4ac-d7e3-4bf1-9d32-286529be0123-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:35.901825 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.901828    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5505e4ac-d7e3-4bf1-9d32-286529be0123-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:35.902054 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:35.901838    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj2jq\" (UniqueName: \"kubernetes.io/projected/5505e4ac-d7e3-4bf1-9d32-286529be0123-kube-api-access-gj2jq\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:36.265385 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.265346    2576 generic.go:356] "Generic (PLEG): container finished" podID="5505e4ac-d7e3-4bf1-9d32-286529be0123" containerID="b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3" exitCode=0
Sep 06 14:32:36.265385 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.265388    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"
Sep 06 14:32:36.265614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.265436    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" event={"ID":"5505e4ac-d7e3-4bf1-9d32-286529be0123","Type":"ContainerDied","Data":"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"}
Sep 06 14:32:36.265614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.265479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z" event={"ID":"5505e4ac-d7e3-4bf1-9d32-286529be0123","Type":"ContainerDied","Data":"4b5b30c19872212d66d2cef12686614d051e241f6c507f402e75a19c68d40285"}
Sep 06 14:32:36.265614 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.265499    2576 scope.go:117] "RemoveContainer" containerID="b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"
Sep 06 14:32:36.274351 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.274333    2576 scope.go:117] "RemoveContainer" containerID="b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"
Sep 06 14:32:36.274657 ip-10-0-133-52 kubenswrapper[2576]: E0906 14:32:36.274637    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3\": container with ID starting with b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3 not found: ID does not exist" containerID="b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"
Sep 06 14:32:36.274738 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.274665    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3"} err="failed to get container status \"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3\": rpc error: code = NotFound desc = could not find container \"b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3\": container with ID starting with b89066cc2b75b86ce4195cf1654a4e381c89ce4796e0b006345b3b52bc4554d3 not found: ID does not exist"
Sep 06 14:32:36.284267 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.284238    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:32:36.287640 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:36.287619    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fda7e6e6-kserve-587696ffdc-t7v6z"]
Sep 06 14:32:37.024952 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.024894    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" podUID="b71212f4-5905-4cb6-8fd9-b43fe1166804" containerName="main" containerID="cri-o://396182e5b8c9d7667ff720af6bfcf019b6b8d018329316a7d045c24eaa1f5a63" gracePeriod=45
Sep 06 14:32:37.272245 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.272211    2576 generic.go:356] "Generic (PLEG): container finished" podID="b71212f4-5905-4cb6-8fd9-b43fe1166804" containerID="396182e5b8c9d7667ff720af6bfcf019b6b8d018329316a7d045c24eaa1f5a63" exitCode=0
Sep 06 14:32:37.272371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.272277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" event={"ID":"b71212f4-5905-4cb6-8fd9-b43fe1166804","Type":"ContainerDied","Data":"396182e5b8c9d7667ff720af6bfcf019b6b8d018329316a7d045c24eaa1f5a63"}
Sep 06 14:32:37.272371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.272318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p" event={"ID":"b71212f4-5905-4cb6-8fd9-b43fe1166804","Type":"ContainerDied","Data":"9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863"}
Sep 06 14:32:37.272371 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.272333    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6e5d4caf559455cd540a802c44752115f6ddcd78b944f61c649d871c5e3863"
Sep 06 14:32:37.279102 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.279042    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:32:37.415781 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415723    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.415979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.415979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415830    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.415979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415855    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.415979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl6r9\" (UniqueName: \"kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.415979 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.415958    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir\") pod \"b71212f4-5905-4cb6-8fd9-b43fe1166804\" (UID: \"b71212f4-5905-4cb6-8fd9-b43fe1166804\") "
Sep 06 14:32:37.416297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.416113    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache" (OuterVolumeSpecName: "model-cache") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:37.416297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.416124    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home" (OuterVolumeSpecName: "home") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:37.416297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.416203    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-home\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.416297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.416223    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-model-cache\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.416297 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.416239    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:37.417978 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.417930    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm" (OuterVolumeSpecName: "dshm") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 14:32:37.418088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.417978    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 14:32:37.418088 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.418017    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9" (OuterVolumeSpecName: "kube-api-access-tl6r9") pod "b71212f4-5905-4cb6-8fd9-b43fe1166804" (UID: "b71212f4-5905-4cb6-8fd9-b43fe1166804"). InnerVolumeSpecName "kube-api-access-tl6r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 14:32:37.516812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.516775    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-dshm\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.516812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.516805    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71212f4-5905-4cb6-8fd9-b43fe1166804-tls-certs\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.516812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.516818    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6r9\" (UniqueName: \"kubernetes.io/projected/b71212f4-5905-4cb6-8fd9-b43fe1166804-kube-api-access-tl6r9\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.517052 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.516828    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71212f4-5905-4cb6-8fd9-b43fe1166804-tmp-dir\") on node \"ip-10-0-133-52.ec2.internal\" DevicePath \"\""
Sep 06 14:32:37.609913 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:37.609835    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5505e4ac-d7e3-4bf1-9d32-286529be0123" path="/var/lib/kubelet/pods/5505e4ac-d7e3-4bf1-9d32-286529be0123/volumes"
Sep 06 14:32:38.276418 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:38.276385    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"
Sep 06 14:32:38.293661 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:38.293629    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:32:38.294953 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:38.294930    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fda7e6e6-kserve-67d86ff755-kqb5p"]
Sep 06 14:32:39.609883 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:39.609849    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71212f4-5905-4cb6-8fd9-b43fe1166804" path="/var/lib/kubelet/pods/b71212f4-5905-4cb6-8fd9-b43fe1166804/volumes"
Sep 06 14:32:43.563382 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:43.563357    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:32:43.705233 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:43.705204    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ntgbk_079a9bd2-3476-4b43-8f7f-25eaa03baabf/ovn-acl-logging/0.log"
Sep 06 14:32:43.960456 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:43.960426    2576 scope.go:117] "RemoveContainer" containerID="50dc3a522b246a78db641fec3697a24a4bf4fcdb173291f8f1b2a1afc67d5afd"
Sep 06 14:32:43.968352 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:43.968328    2576 scope.go:117] "RemoveContainer" containerID="5dbaebdc61d4cec7cbeb53f6a06a5a585f7669cb22a0b2bb95b2065075e8b1f3"
Sep 06 14:32:43.975747 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:32:43.975730    2576 scope.go:117] "RemoveContainer" containerID="4fb261a1f32bb73b418b729832eb5e46532f1e17d83d4c670c8421a8177e261c"
Sep 06 14:33:26.274182 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:26.274149    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9285d_9e8488b3-e109-41ae-acb2-077daf435dfc/server/0.log"
Sep 06 14:33:26.605077 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:26.604995    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-w7ptw_ada67cdc-a7f0-4d66-b335-ac99cdf322d0/copy-manifests/0.log"
Sep 06 14:33:26.611393 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:26.611364    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-w7ptw_ada67cdc-a7f0-4d66-b335-ac99cdf322d0/manager/0.log"
Sep 06 14:33:27.260233 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:27.260205    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29811735-6ncxt_3b3869b8-f100-4573-ae50-8028daa257a7/cleanup/0.log"
Sep 06 14:33:27.359294 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:27.359262    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29811750-cmrbc_c4eb07e4-0987-4b47-a632-7c5c4317ffc7/cleanup/0.log"
Sep 06 14:33:29.229055 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:29.229026    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw_ad195460-b4fc-49e1-af85-2b9a9e228fca/istio-proxy/0.log"
Sep 06 14:33:29.553671 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:29.553590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-wz2lc_b02edfcb-2107-4b0c-a061-91ef848f3dc6/kube-auth-proxy/0.log"
Sep 06 14:33:30.444491 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:30.444453    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lghxs_f18d79ee-2721-4abf-80e6-28b4baebfecc/main/0.log"
Sep 06 14:33:30.550653 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:30.550620    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-5s5bg_6919c6fe-8749-4d52-a7de-d5c298eb5b40/main/0.log"
Sep 06 14:33:30.981486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:30.981458    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-zjlzp_37ab3082-1fb0-43c5-8ff1-18881db755b7/main/0.log"
Sep 06 14:33:36.649394 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649308    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfljt/must-gather-8vbfv"]
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649590    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5505e4ac-d7e3-4bf1-9d32-286529be0123" containerName="main"
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649602    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5505e4ac-d7e3-4bf1-9d32-286529be0123" containerName="main"
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649623    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71212f4-5905-4cb6-8fd9-b43fe1166804" containerName="main"
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649628    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71212f4-5905-4cb6-8fd9-b43fe1166804" containerName="main"
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649674    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5505e4ac-d7e3-4bf1-9d32-286529be0123" containerName="main"
Sep 06 14:33:36.649813 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.649688    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71212f4-5905-4cb6-8fd9-b43fe1166804" containerName="main"
Sep 06 14:33:36.652717 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.652697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.654950 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.654929    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"kube-root-ca.crt\""
Sep 06 14:33:36.655037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.654936    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfljt\"/\"default-dockercfg-9w4wc\""
Sep 06 14:33:36.655037 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.654974    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfljt\"/\"openshift-service-ca.crt\""
Sep 06 14:33:36.660250 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.660229    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/must-gather-8vbfv"]
Sep 06 14:33:36.757634 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.757590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-must-gather-output\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.757854 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.757654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgzj\" (UniqueName: \"kubernetes.io/projected/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-kube-api-access-flgzj\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.858806 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.858748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-must-gather-output\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.858986 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.858823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgzj\" (UniqueName: \"kubernetes.io/projected/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-kube-api-access-flgzj\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.859145 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.859119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-must-gather-output\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.868225 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.868202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgzj\" (UniqueName: \"kubernetes.io/projected/5dbdaa28-a2b7-4d2e-9af1-31e880780b2e-kube-api-access-flgzj\") pod \"must-gather-8vbfv\" (UID: \"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e\") " pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:36.961999 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:36.961965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/must-gather-8vbfv"
Sep 06 14:33:37.234657 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:37.234577    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/must-gather-8vbfv"]
Sep 06 14:33:37.237905 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:33:37.237879    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dbdaa28_a2b7_4d2e_9af1_31e880780b2e.slice/crio-8f04d763277b6c8b9d834497f92392ba90b69853ebae912473c77c11c29b6098 WatchSource:0}: Error finding container 8f04d763277b6c8b9d834497f92392ba90b69853ebae912473c77c11c29b6098: Status 404 returned error can't find the container with id 8f04d763277b6c8b9d834497f92392ba90b69853ebae912473c77c11c29b6098
Sep 06 14:33:37.240162 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:37.240145    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 14:33:37.521332 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:37.521246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/must-gather-8vbfv" event={"ID":"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e","Type":"ContainerStarted","Data":"8f04d763277b6c8b9d834497f92392ba90b69853ebae912473c77c11c29b6098"}
Sep 06 14:33:38.529640 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:38.529577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/must-gather-8vbfv" event={"ID":"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e","Type":"ContainerStarted","Data":"8b9ef7ca4008a342b058f2e2dac6e48841f749983f3bdd23cbdae05fb7ae10e7"}
Sep 06 14:33:38.529640 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:38.529644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/must-gather-8vbfv" event={"ID":"5dbdaa28-a2b7-4d2e-9af1-31e880780b2e","Type":"ContainerStarted","Data":"0c1a4aef665f377b98b4d30aefb813bcf2362dddb9fc4f0989d7fd5ba2f32272"}
Sep 06 14:33:38.545210 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:38.545149    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfljt/must-gather-8vbfv" podStartSLOduration=1.7930657060000001 podStartE2EDuration="2.545128537s" totalImagesPullingTime="752.062831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 14:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:33:38.543240905 +0000 UTC m=+2155.569306081" watchObservedRunningTime="2026-09-06 14:33:38.545128537 +0000 UTC m=+2155.571193715"
Sep 06 14:33:39.524928 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:39.524895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nmr22_0a218de8-5016-4c86-971d-3f14763198c1/global-pull-secret-syncer/0.log"
Sep 06 14:33:39.611228 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:39.611184    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nsfxb_6de74b58-56f8-4cb2-9823-7f4034db8d3d/konnectivity-agent/0.log"
Sep 06 14:33:39.672819 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:39.672789    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-52.ec2.internal_c714293c914577f6a860a52a7e5762b6/haproxy/0.log"
Sep 06 14:33:46.442452 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:46.442415    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ht946_5be1a55b-675c-4f8f-bae5-20aa46ef7298/node-exporter/0.log"
Sep 06 14:33:46.461153 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:46.461120    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ht946_5be1a55b-675c-4f8f-bae5-20aa46ef7298/kube-rbac-proxy/0.log"
Sep 06 14:33:46.479486 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:46.479448    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ht946_5be1a55b-675c-4f8f-bae5-20aa46ef7298/init-textfile/0.log"
Sep 06 14:33:49.771243 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.771201    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"]
Sep 06 14:33:49.777258 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.777226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.784780 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.784020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"]
Sep 06 14:33:49.886191 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.886141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkfdt\" (UniqueName: \"kubernetes.io/projected/5446b6e1-2762-4dcd-8f8a-238724906054-kube-api-access-gkfdt\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.886386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.886218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-sys\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.886386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.886272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-podres\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.886386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.886297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-lib-modules\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.886386 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.886337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-proc\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.986812 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.986751    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkfdt\" (UniqueName: \"kubernetes.io/projected/5446b6e1-2762-4dcd-8f8a-238724906054-kube-api-access-gkfdt\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.986823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-sys\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.986876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-podres\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.986902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-lib-modules\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987019 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.986940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-proc\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.987033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-proc\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.987061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-podres\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.987069    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-lib-modules\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.987222 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.987121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5446b6e1-2762-4dcd-8f8a-238724906054-sys\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:49.996339 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:49.994484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkfdt\" (UniqueName: \"kubernetes.io/projected/5446b6e1-2762-4dcd-8f8a-238724906054-kube-api-access-gkfdt\") pod \"perf-node-gather-daemonset-5ntrf\" (UID: \"5446b6e1-2762-4dcd-8f8a-238724906054\") " pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:50.116227 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.116129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:50.408282 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.408152    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"]
Sep 06 14:33:50.412699 ip-10-0-133-52 kubenswrapper[2576]: W0906 14:33:50.412658    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5446b6e1_2762_4dcd_8f8a_238724906054.slice/crio-a85e5d8926f9287b37e2536a23fb4477de276f892e8d825183e813537012de42 WatchSource:0}: Error finding container a85e5d8926f9287b37e2536a23fb4477de276f892e8d825183e813537012de42: Status 404 returned error can't find the container with id a85e5d8926f9287b37e2536a23fb4477de276f892e8d825183e813537012de42
Sep 06 14:33:50.613387 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.613349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf" event={"ID":"5446b6e1-2762-4dcd-8f8a-238724906054","Type":"ContainerStarted","Data":"9cca95dac4345e1fc8a222f145d2cc2f1a8da1b188e1c158b595b9e96a25904b"}
Sep 06 14:33:50.613559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.613412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf" event={"ID":"5446b6e1-2762-4dcd-8f8a-238724906054","Type":"ContainerStarted","Data":"a85e5d8926f9287b37e2536a23fb4477de276f892e8d825183e813537012de42"}
Sep 06 14:33:50.613559 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.613486    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:50.629970 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:50.629909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf" podStartSLOduration=1.629887895 podStartE2EDuration="1.629887895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 14:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 14:33:50.628287568 +0000 UTC m=+2167.654352745" watchObservedRunningTime="2026-09-06 14:33:50.629887895 +0000 UTC m=+2167.655953066"
Sep 06 14:33:51.571470 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:51.571439    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kv2lw_45826550-7beb-46ac-9934-421749d069e2/dns/0.log"
Sep 06 14:33:51.589326 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:51.589295    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kv2lw_45826550-7beb-46ac-9934-421749d069e2/kube-rbac-proxy/0.log"
Sep 06 14:33:51.724528 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:51.724495    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l95xv_bae7c569-ded6-42e5-9ee9-4c499a27b95b/dns-node-resolver/0.log"
Sep 06 14:33:52.252823 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:52.252792    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-97f89fc94-tvg8r_50e352b8-442e-45f2-96ac-5ab8567066b7/registry/0.log"
Sep 06 14:33:52.296997 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:52.296965    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dzd9g_fe20f3c5-a2b7-488d-af61-b5845188ff63/node-ca/0.log"
Sep 06 14:33:53.207689 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:53.207659    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw_ad195460-b4fc-49e1-af85-2b9a9e228fca/istio-proxy/0.log"
Sep 06 14:33:53.778748 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:53.778671    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-wz2lc_b02edfcb-2107-4b0c-a061-91ef848f3dc6/kube-auth-proxy/0.log"
Sep 06 14:33:54.780359 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:54.780314    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbkk8_d3fe2a27-6404-4a6f-8f57-65a7ac100ee9/serve-healthcheck-canary/0.log"
Sep 06 14:33:55.259296 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:55.259268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5psgm_bfbcd790-7eba-4fa1-8e7d-0904c59de236/kube-rbac-proxy/0.log"
Sep 06 14:33:55.276739 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:55.276702    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5psgm_bfbcd790-7eba-4fa1-8e7d-0904c59de236/exporter/0.log"
Sep 06 14:33:55.300467 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:55.295676    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5psgm_bfbcd790-7eba-4fa1-8e7d-0904c59de236/extractor/0.log"
Sep 06 14:33:56.628200 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:56.628172    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfljt/perf-node-gather-daemonset-5ntrf"
Sep 06 14:33:57.705436 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:57.705398    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9285d_9e8488b3-e109-41ae-acb2-077daf435dfc/server/0.log"
Sep 06 14:33:57.892651 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:57.892611    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-w7ptw_ada67cdc-a7f0-4d66-b335-ac99cdf322d0/manager/0.log"
Sep 06 14:33:57.909244 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:57.909194    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6858f974-w7ptw_ada67cdc-a7f0-4d66-b335-ac99cdf322d0/copy-manifests/0.log"
Sep 06 14:33:59.480621 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:33:59.480523    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-849f85686b-jlfx4_221bda4b-5512-4144-a306-f1411b81f925/manager/0.log"
Sep 06 14:34:00.169198 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:00.169161    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw_ad195460-b4fc-49e1-af85-2b9a9e228fca/istio-proxy/0.log"
Sep 06 14:34:00.723099 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:00.723052    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-wz2lc_b02edfcb-2107-4b0c-a061-91ef848f3dc6/kube-auth-proxy/0.log"
Sep 06 14:34:02.385488 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:02.385447    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jnxzw_ad195460-b4fc-49e1-af85-2b9a9e228fca/istio-proxy/0.log"
Sep 06 14:34:02.953255 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:02.953210    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86996bf9f5-wz2lc_b02edfcb-2107-4b0c-a061-91ef848f3dc6/kube-auth-proxy/0.log"
Sep 06 14:34:05.900372 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.900334    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/kube-multus-additional-cni-plugins/0.log"
Sep 06 14:34:05.917041 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.917007    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/egress-router-binary-copy/0.log"
Sep 06 14:34:05.934012 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.933970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/cni-plugins/0.log"
Sep 06 14:34:05.951724 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.951697    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/bond-cni-plugin/0.log"
Sep 06 14:34:05.968835 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.968803    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/routeoverride-cni/0.log"
Sep 06 14:34:05.985452 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:05.985417    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/whereabouts-cni-bincopy/0.log"
Sep 06 14:34:06.002805 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:06.002752    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f4xxx_2f0bd173-2c38-4019-9a81-84cb8910cd0c/whereabouts-cni/0.log"
Sep 06 14:34:06.239434 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:06.239394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hhjln_c9733cc8-58c9-4ca6-beb0-69dcf9b0a04a/kube-multus/0.log"
Sep 06 14:34:06.333477 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:06.333449    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldcbb_f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7/network-metrics-daemon/0.log"
Sep 06 14:34:06.346363 ip-10-0-133-52 kubenswrapper[2576]: I0906 14:34:06.346331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ldcbb_f5cc8ab5-9e3c-4e89-81a5-c56822e7bbf7/kube-rbac-proxy/0.log"